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Preface 


The  structure  of  this  document  will  support  two 
categories  of  readers:  a)  senior  management  personnel 
Interested  In  key  Issues,  findings,  and  recommendations,  and 
b)  acquisition  personnel.  Senior  managers  should  read 
Chapter  I,  Introduction,  and  Chapter  V,  Recommendations. 

The  remainder  of  the  acquisition  work  force  should  read  all 
five  chapters. 

Chapter  I  Introduces  the  topic  and  Identifies  the 
problem.  Chapter  II  reviews  the  available  literature  on 
award  fee  contracts  and  software  acquisition.  Chapter  III 
covers  the  research  methodology  used  for  this  study. 

Chapter  IV  siunmarlzes  the  data  obtained  during  the 
Interviews.  Chapter  V  summarizes  the  findings  of  this 
research  and  proposes  recommendations  for  further  research 
to  Improve  the  system  progreun  offices  ability  to  effectively 
use  award  fee  contracts  based  on  the  data  obtained  during 
the  literature  review  and  Interviews. 

A  number  of  Individuals  provided  guidance  and  support 
to  the  researcher.  These  Include:  my  thesis  advisors,  Mr. 
Daniel  V.  Ferens  and  Maj  Dennis  L.  Hull;  my  thesis  sponsor, 
HQ  AFIC/LEC,  Mr.  David  Hood. 

This  thesis  Is  dedicated  to  my  wife,  Charlotte  Ellen 
Hunter,  without  whose  help  and  support  this  document  would 
never  have  been  completed. 
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Abstract 

The  focus  of  this  research  effort  was  to  determine.  In 
general  terms,  what  contractual  tools  are  available  to 
Influence  contractor  performance  during  the  software 
development  process  and,  specifically,  whether  or  not  award 
fee  contracts  are  appropriate  tools.  First,  a  qualitative, 
in  depth  literature  review  was  done.  Next,  personal 
interviews  were  conducted  with  Program  Directors  and  Deputy 
Program  Directors  at  Aeronautical  Systems  Center,  Electronic 
Systems  Center,  and  Space  and  Missiles  Systems  Center. 

Then,  after  the  results  were  summarized,  the  data  was 
analyzed  and  findings  were  made. 

Findings  Include:  (1)  award  fee  was  found  to  be  the 
most  flexible  provision  in  the  FAR  and  its  supplements  to 
influence  contractor  performance  during  the  software 
development  process;  (2)  the  literature  search  found  a 
strong  consensus  on  major  events  or  milestones  in  the 
software  development  process;  and,  (3)  program  requirements 
are  the  key  factor  that  drives  the  linkage  between  award  fee 
and  the  software  development  events.  For  software 
development,  award  fee  contracts  give  the  government  a 
flexible  and  effective  means  to  effect  the  software 
development  process. 
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AWARD  FEE  IN  SOFTWARE  ACQUISITION 


I .  Introduction 


When  the  1980  Five  Year  Defense  Plan  Is  compared  with 

the  1992  Six  Year  Defense  Plan,  It  appears  that  the  defense 

budget  Is  shrinking.  This  means  there  will  be  less  money 

available  for  development  and  maintenance  of  current  and 

future  weapon  systems.  Many  of  today's  weapon  systems 

contain  large  amounts  of  software.  Software  development  Is 

fast  becoming  the  critical  part  of  all  weapon  systems 

development  and  maintenance. 

Software  often  Incurs  greater  costs  and  more 
Intractable  problems.  Its  development  and  maintenance 
account  for  as  much  as  10%  of  the  entire  defense 
budget,  ...  with  80%  of  that  figure  going  towards 
labor-intensive  rework  and  updating.  (Goldberg, 
1990:60) 

If  the  current  state  of  the  hardware  development  Is  compared 
with  that  of  software,  hardware  Is  keeping  pace  with  the 
technological  advances  while  "software  systems  lag  far 
behind"  (Goldberg,  1990:60). 

Government  and  contractors  alike  are  aware  of  the 
problems  associated  with  development  and  maintenance  of 
software.  To  eliminate  the  lack  of  visibility  In  the 
software  development  process,  the  Department  of  Defense 
(DoD)  developed  DoD-STD-2167A,  Defense  Systems  Software 
Development.  This  document  describes  a  structured  approach 
to  software  development.  It  outlines  the  development 
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process  and  describes  the  required  documentation.  However, 
this  standard  and  others  have  still  been  inadequate  to 
facilitate  the  development  of  software  within  cost  and 
schedule  (Goldberg,  1990:60). 


General  Issue 


According  to  Goldberg,  there  is  a  need  to  develop  an 
environment  where  both  the  Government  and  contractor  develop 
supportable  and  maintainable  weapon  systems  software  on  time 
and  within  budget.  If  this  is  done,  there  will  be  fewer 
programs  with 

Intractable  software  problems...  [leading  to] 
...canceled  programs  (e.g.,  DIVAD),  or  rather  dubious 
circumventions  of  specifications,  as  in  the  B-IB 
bomber.  Aegis  shipboard  air  defense  system,  and  M-x 
missile  guidance  system.  (Goldberg,  1990:60) 


There  needs  to  be  a  way  to  focus  both  Government  and 
contractor  attention  on  the  software  development  process 
that  is  both  flexible  and  contractually  binding.  This  would 
allow  the  Government  to  better  assess  the  contractor's 
performance  during  the  development  process. 


Background 

"A  major  problem  area  quickly  becoming  evident 
...[is]...  software  management.  Software  is  the  least 
understood  and  highest  risk  of  a  typical  program"  (DSMC, 
1989:23).  Costs  associated  with  hardware  and  software 
development  are  a  small  part  of  the  total  life  cycle  cost. 
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Weapon  systems  maintenance  costs  comprise  the  largest  part 
of  life  cycle  cost.  Figure  1,  Normal  Cost  Distribution  of  a 
Typical  DoD  Program,  shows  the  relative  distribution  of 
costs  per  development  phase.  This  figure  divides  the  phases 
into  system  research  and  development,  production,  and 
operation  and  support.  The  chart  in  Figure  1  indicates  that 
the  operations  and  support  phase  accounts  for  60  percent  of 
the  total  life  cycle  cost.  Weapon  systems  life  cycle  cost 
is  a  complex  issue  and  is  outside  the  scope  of  this 


Figure  1.  Normal  Cost  Distribution  of  a  Typical  DoD 


Program  (DSMC,  1986:17) 

research . 

By  the  start  of  production,  85  percent  of  the  life 
cycle  costs  are  fixed.  This  is  what  makes  software 
maintenance  costs  so  high.  Figure  2,  Effect  of  Early 
Decisions  on  Life  Cycle  Cost,  shows  the  effects  early 
development  decisions  make  on  life  cycle  costs.  There  are 
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three  key  products  which  are  essential  for  software 
maintenance  that  come  from  the  software  development  process: 
documentation,  modifiable  code,  and  source  code.  The 
decisions  on  the  quality  and  completeness  of  these  tools  are 
decided  during  development,  where  the  pressures  of 
development  cost  and  schedule  may  sometimes  override 
concerns  for  maintenance. 

Figure  3,  Fault  Removal  Cost,  shows  the  relationship 
between  correcting  software  and  documentation  early  In 
development  and  later  In  operation.  The  costs  to  fix  faults 
during  the  operation  and  support  phase  are  over  80  times 
more  expensive  than  during  development  ( Integrated  Computer 

Too 
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Figure  2.  Effect  of  Early  Decisions  on  Life  Cycle  Cost 
(DSMC,  1986:17-3) 
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Systems,  1988:340-1-27).  The  upper  limit  is  unknown,  for 
there  are  too  many  factors  to  calculate  a  true  bound. 


®  20 


Requirements  Code  /  Unit  Test  System  Test 

Design  integration  Test  Operation 

Phase  of  Life  Cycle 


Figure  3.  Fault  Removal  Cost  (Integrated  Computer  Systems, 


1988:340-1-27) 

The  cost  of  software  development  projects  has  Increased 
since  the  1960 's  with  an  increase  in  the  contribution  from 
software.  In  the  past,  hardware  was  the  major  cost  driver 
for  weapon  system  development.  Today,  software  is  the  major 
cost  driver.  Figure  4  graphically  displays  this 
relationship. 

With  these  facts,  a  means  of  influencing  the  software 
development  process  needs  to  be  found.  The  development 
process  must  be  monitored  regularly,  contractor  performance 
must  be  evaluated,  and  the  Government  must  have  a  feedback 
mechanism  to  show  the  contractor  what  constitutes  acceptable 
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— HARDWARE  -A-  SOFTWARE 


Figure  4.  A  Comparison  Between  Software  and  Hardware 
Cost  for  1960  to  1985  (Simpson,  1987:3). 

performance . 

Problem  Statement 

The  Government  has  limited  contractual  tools  available 
during  the  software  development  process  to  provide  the 
contractor  with  feedback  and  to  Insure  the  contractor 
delivers  software  that  Is  on  schedule,  within  cost,  and 
maintainable.  The  research  question  Is:  What  contractual 
tools  are  available  to  Influence  contractor  performance 
during  the  software  development  process? 
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Investigative  Questions 


To  determine  what  contractual  tools  are  available  to 
Influence  contractor  performance  during  the  development 
process,  the  following  questions  will  be  researched: 

1.  What  provisions  are  available  In  the  Federal 
Acquisition  Regulation  (FAR)  and  Its  supplemental 
regulations  to  Influence  contractor  performance 
during  the  development  process? 

2.  What  events  or  milestones  in  the  development  process 
should  be  used  to  evaluate  contractor  performance 
during  the  software  development  process? 

3.  How  should  award  fee,  as  described  In  the  FAR  and  Its 
supplements,  and  software  development  events  be 
linked  to  Influence  contractor  performance? 

4.  Can  the  award  fee  process  be  used  to  Influence 
contractor  performance  during  software  development? 

5.  What  award  fee  criteria  should  be  used  to  evaluate 
contractor  performance? 

6.  Does  the  award  fee  process  Include  an  appropriate 
type  and  form  of  feedback  to  give  to  the  contractor 
for  Improving  his  ability  to  develop  maintainable 
software  on  time,  and  within  cost? 


Scope  and  Limitations 

This  thesis  will  examine  only  a  few  of  the  numerous 
approaches  available  to  Influence  the  software  development 
process.  Approaches  not  covered  include  tracking  software 
metrics,  growth  In  lines  of  code,  and  code  and  unit  test 
deficiencies.  The  research  starts  with  the  premise  of  using 
the  FAR  and  Its  supplements  as  the  primary  tool  to  Insure 
quality  software  Is  delivered  to  the  Government. 
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Summary 


This  chapter  presented  a  brief  summary  of  the  future 
financial  picture  of  the  defense  budget  and  problems 
associated  with  software  development.  It  discussed  the  need 
for  developing  tools  to  improve  the  software  development 
process  because,  as  more  modern  weapons  systems  are 
developed,  the  eunount  and  complexity  of  weapon  systems 
software  will  continue  to  grow. 

Chapter  II  reviews  current  literature  on  software 
development.  The  chapter  discusses  the  contractual 
requirements  and  software  development  process,  reviews  the 
tools  available  in  the  FAR  and  its  supplements,  and 
evaluates  suitability  of  existing  methods  to  influence 
contractor  performance.  The  software  development  process  is 
described  and  the  various  events  and  milestones  suitable  for 
linkage  to  the  contractual  requirements  are  reviewed. 
Contractor  evaluation  criteria  and  processes  are  evaluated. 
Finally,  types  and  forms  of  contractor  feedback  are  analyzed 
to  determine  their  relative  merits. 
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II .  Literature  Review 


Introduction 

This  chapter  summarizes  the  qualitative  Information 
available  for  possible  answers  to  the  questions  proposed  In 
Chapter  I.  The  qualitative  Information  available  on  each 
question  Is  reviewed  followed  by  a  research  summary. 

Question  1.  What  provisions  are  available  In  the  Federal 
Acquisition  Regulation  (FAR)  and  Its  supplemental 
regulations  to  Influence  contractor  performance  during  the 
development  process? 

The  FAR  describes  several  types  of  contracts.  The 
basic  types  are  Fixed  Price  (FP).  and  Cost  Reimbursement 
(CP).  In  a  Fixed  Price  type  of  contract,  the  contractor 
has  agreed  to  a  fixed-price  for  the  work  to  be  performed. 

If  the  contractor  exceeds  the  fixed  price,  any  additional 
cost  is  his  responsibility.  In  a  Cost  Reimbursement 
contract,  the  government  will  pay  all  allowable  and 
allocable  costs  associated  with  the  contract  up  to  an  agreed 
to  celling  (TASC,  undated:  Book  2,  E-1  to  E-6;  FAR, 

1990: Part  2). 

Both  FP  and  CP  contracts  can  be  varied  by  the  type  of 
fee  arrangement  In  the  contract.  The  eunount  of  fee  or 
profit  the  contractor  receives  may  be  structured  to  consider 
several  factors;  for  example,  performance,  quality  of  work, 
schedule,  and  maintainable  software. 
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The  types  of  fee  arrangements  include  fixed  fee  (FF), 
incentive  fee  (IF),  and  award  fee  (AF).  A  fixed  fee 
arrangement  is  set  at  the  start  of  the  contract  and  is  used 
primarily  when  the  item  being  purchased  is  not  defined  well. 
Incentive  fee  arrangements  vary  significantly  and  may 
Include  schedule  and  technical  performance  incentives  in 
addition  to  cost  incentives.  An  award  fee  is  used  when  the 
government  wants  a  contractor  to  produce  results,  such  as 
acceptable  documentation  or  maintainable  software,  with 
standards  which  cannot  be  objectively  quantified. 

Possible  combinations  of  contract  types  with  different 
fee  arrangements  are  depicted  in  Table  1.  The  type  of 
contract  used  should  be  selected  to  fit  the  specific 
requirements  of  the  weapon  system  being  developed.  For 
example,  a  FFP  contract  would  be  used  by  the  government  to 
contract  of  a  lot  of  500  AM-9  Sidewinder  air-to-air  missies. 
On  the  other  hand,  the  development  of  the  F-22  fighter 
aircraft  would  use  a  CP  contract  because  the  government 
share  of  risk  in  a  new  development  is  greater. 

The  FAR  and  its  supplements  have  other  tools,  which  are 
standard  clauses  required  in  all  contracts,  that  can  be  used 
to  influence  contractor  performance.  One,  show  cause 
notice,  is  used  "when  the  contractor  has  defaulted  by 
failure  to  make  delivery  of  the  supplies  or  to  perform  the 
services  within  the  specified  time”  (FAR,  1992:49. 402-3(c) ) . 
The  government  uses  a  show  cause  notice  to  "call  the 
contractor's  attention  to  the  contractual  liabilities  if  the 
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contract  Is  terminated  for  default,  and  request  the 
contractor  to  show  cause  why  the  contract  should  not  be 
terminated  for  default"  (FAR,  1992:49. 402-3(e) ( 1) ) . 

TABLE  1 

Combinations  of  Contract  Types  and  Fee  Arrangements 


Contract  Type  Fee  Tvoe  Contract/Fee 


CP 

IF 

CPIF 

CP 

AF 

CPAF 

CP 

IF/AF 

CPIF/AF 

CP 

FF 

CPFF 

FP 

IF 

FPIF 

FP 

AF 

FPAF 

FP 

IF/AF 

FPIF/AF 

FP 

FF 

FPFF 

Another  tool  is  to  withhold  payments.  The  Defense 
Federal  Acquisition  Regulation  (DFAR)  contains  a  provision 
covering  remedies  for  noncomplying  technical  data  (DFAR, 
1992:227.403-74).  This  provision  allows  the  government  to 
withhold  payment  if  technical  data  is  not  delivered  in  a 
timely  manner.  The  amount  of  payment  withheld  is  usually 
based  on  the  "estimated  value  of  the  technical  data  to  the 
government"  (DFAR,  1992:227.403-74). 

The  most  drastic  is  the  termination  for  default  clause 
which  permits  the  government  to  terminate  the  contract 
reprocure  the  orginal  supplies  or  services  and  charge  the 
defaulted  contractor  with  any  excess  reprocurement  costs. 
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These  are  but  three  of  the  more  drastic  measures  to 


modify  contractor  performance,  and  all  such  measures  are 
subject  to  the  disputes  clause  In  the  FAR  and  Its 
supplements . 

Question  1  Summary.  The  type  of  contract  used  is  a  key 
FAR  management  tool  available  to  Influence  contractor 
performance  during  the  software  development  process.  In 
this  review,  award  fee  Is  the  most  flexible  provision  In  the 
FAR  and  Its  supplements  to  Influence  contractor  performance 
during  the  development  process.  Award  fee  also  provides  an 
appropriate  vehicle  for  feedback. 

Question  2.  What  events  .or  milestones  in  the  development 
process  should  be  used  to  evaluate  contractor  performance 
during  the  software  development  process? 

There  are  many  articles  and  books  on  events  and 
milestones  In  the  software  development  process.  This 
section  reviews  approaches  of  several  authors  and  summarizes 
their  findings. 

Today,  DoD-STD-2167A,  Defense  System  Software 
Development .  and  Defense  Systems  Management  College  (DSMC)'s 
Mission  Critical  Computer  Resources  fMCCR)  Management  Guide 
are  handy  reference  materials  for  any  government  project 
office  performing  the  task  of  software  development.  However 
before  the  first  of  these  documents  was  ever  published,  Lt 
Col  Willard  presented  a  management  concept  that  can  be  used 
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to  evaluate  contractor  software  development  performance  that 
differed  little  from  the  first  DSMC  Systems  Engineering 
Management  fSEM^  Guide.  Lt  Col  Willard's  concept  is  divided 
into  four  main  areas:  software  development  plan, 
development  specification,  personnel,  and  reviews  and 
testing. 

The  software  development  plan  is  developed  by  the 

contractor  and  defines  how  the  contractor  will  accomplish 

the  software  development  activity.  This  plan  should 

accomplish  three  major  areas  as  a  minimum.  One,  provide 

policies  and  procedures  that  have  clear  visibility  into 

software  development,  integration,  and  test.  Two,  reduce 

the  risk  of  schedule  delays  and  cost  overruns.  Three, 

provide  for  better  communication  between  the  government  and 

the  contractor.  (Willard,  1985:9) 

After  the  plan  is  developed,  a  development 

specification  is  "produced  and  validated  prior  to  writing 

any  software"  (Willard,  1985:9).  The  author  says. 

Any  additional  time  spent  in  project  definition  will  be 
amply  repaid  later  because  good  functional  descriptions 
will  allow  better  costing  and  scheduling,  and  the 
software  development  will  stand  a  better  chance  of 
meeting  desired  performance  standards.  (Willard, 
1985:9) 

The  author's  concerns  with  personnel  center  around  the 
contractor's  ability  to  man  the  software  development  program 
with  trained  and  qualified  personnel.  These  concerns 
Increase  if  the  development  effort  is  two  or  more  years  in 
length.  For  these  longer  efforts,  Lt  Col  Willard  believes 
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"personnel  turnover  and  changes  in  program  management 
direction  must  be  anticipated  and  reflected  in  the  planning 
documentation”  (Willard,  1985:10). 

The  last  area  is  review  and  testing.  Lt  Col  Willard 
believes  they  must  Include  evaluating  the  proposal, 
establishing  a  project  environment  based  on  appropriate 
reviews,  providing  the  criteria  for  testing  software  against 
the  performance  standards  described  in  the  specifications, 
and  setting  appropriate  milestones  for  review  (Willard, 
1985:10).  The  author  states  "The  government's  evaluation 
procedures  must  be  in  place  both  to  review  any  contractor 
plan  and  insure  that  plans  are  implemented  in  a  timely 
manner"  (Willard,  1985:10). 

Lt  Col  Willard's  paper  outlines  a  concept  for 
controlling  the  software  development  process  and  supplies 
several  areas  that  can  be  used  to  evaluate  contractor 
performance. 

Simpson's  book.  New  Techniques  in  Software  Project 
Management .  describes  a  software  development  life  cycle  with 
four  major  stages.  Each  major  stage  contains  one  to  seven 
minor  activities  building  on  each  other.  The  four  major 
stages  in  Simpson's  life  cycle  are  definition,  design. 
Implementation,  and  maintenance.  Simpson  places  a 
feasibility  study,  requirements  definition,  and  system 
functional  specification  within  the  definition  stage  of  the 
life  cycle  (Simpson,  1987:14).  The  design  stage  normally 
includes  system  design,  program  or  module  design,  and  model 
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development  (Simpson,  1987:14).  Simpson's  implementation 

stage  contains  the  activities  associated  with  software  code 

development.  These  activities  include  module  design/coding, 

test  and  debugging  of  modules,  building  a  skeleton  system, 

system  testing,  and  system  installation  (Simpson,  1987:15). 

This  life  cycle  includes  the  maintenance  stage,  because 

Historically,  it  is  quite  often  the  stage  that  is  most 
often  underestimated  when  resources  and  time  are 
allocated  in  a  software  development  project.  In  fact, 
this  stage  is  quite  often  the  single  largest  component 
of  any  software  project.  (Simpson,  1987:16) 

This  software  development  life  cycle  contains  only  four 

stages,  but  within  each  stage  there  are  events  or  milestones 

that  can  be  used  to  evaluate  a  contractor's  performance. 

DOD-STD-2167A,  Defense  System  Software  Development. 

divides  the  software  development  life-cycle  into  eight 

events  each  leading  up  to  potentially  nine  milestones  (DoD, 

1988:10).  Each  milestone  summarizes  the  contractor's 

analysis  or  design  completed  to  this  point.  The  contractor 

then  presents  his  findings  for  government  review  and 

approval.  The  nine  milestones  are  system  requirements 

review  (SRR),  system  design  review  (SDR),  software 

specification  review  (SSR),  preliminary  design  review  (PDR), 

critical  design  review  (CDR),  test  readiness  review  (TRR), 

functional  configuration  audit  (FCA),  physical  configuration 

audit  (PCA),  and  formal  qualification  review  (FQR)  (DoD, 

1988:10) . 

In  DOD-STD-2167A,  SRR  is  the  first  step  in  the 
development  process  and  is  the  step  where  all  system  level 
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requirements  are  documented.  SRR  Is  also  the  first  attempt 
at  a  functional  baseline  that  the  government  will  approve 
(DoD,  1988:10).  SDR  occurs  when  the  contractor  documents 
the  system  level  design  and  supplies  to  the  government  the 
final  functional  baseline.  Once  the  functional  baseline  Is 
approved,  all  other  development  requirements  will  be  based 
on  this  baseline  (DoD,  1988:12).  SSR  culminates  the 
software  requirements  process  and  documents  the  software 
specifications.  When  the  Software  Requirements 
Specifications  and  associated  Interface  Requirements 
Specification  are  completed  and  approved  by  the  government, 
they  become  the  allocated  baseline  on  which  the  software  Is 
designed  (DoD,  1988:21).  The  PDR  documents  how  software 
requirements  are  allocated  to  computer  software  components. 
This  allows  the  government  the  ability  to  review  and  approve 
the  contractor's  preliminary  software  design  (DoD,  1988:23). 
CDR  documents  the  contractor's  detailed  design  for  all 
computer  software  configuration  Items  (CSCIs).  This  allows 
the  government  to  review  and  approve  the  design  before  the 
contractor  builds  software  (DoD,  1988:25).  TRR  Is  the  event 
at  which  the  government  reviews  the  contractor's  Integration 
test  data  for  each  CSC I  to  see  If  the  contractor  Is  ready 
for  formal  CSCI  Integration  testing.  Formal  testing  will 
not  start  until  the  government  approves  TRR  (DoD,  1988:29). 
During  FCA,  the  government  evaluates  the  software  to  see 
that  each  CSCI  functions  according  to  development 
requirements  (DoD,  1988:31).  PCA  occurs  when  the  government 
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evaluates  the  CSCIs  against  the  design  documentation  to 
insure  each  CSCI  is  documented  correctly  (DoD,  1988:31). 

The  last  review  is  FQR.  This  review  evaluates  the  CSCI 
against  the  allocated  requirement  for  that  item  (DoD/ 
1988:31) . 

These  nine  milestones  allow  the  government  to  evaluate 
the  contractor's  performance  based  on  the  different  stages 
in  the  development  process  (analysis/  design/  code/  unit 
test/  system  integration  test/  and  system  test) . 

Yourdon's  book/  Managing  the  System  Life  Cycle,  covers 
the  total  software  development  process  from  user's 
requirements  to  installation.  The  nine  major  events  in 
Yourdon's  life  cycle  are  survey/  analysis/  design/ 
implementation/  acceptance  test  generation/  .quality 
assurance/  procedure  description/  database  conversion/  and 
installation. 

The  survey  is  described  as  a  "feasibility  study  or 
initial  business  study"  (Yourdon/  1988:54).  The  survey's 
major  purpose  is  to  identify  current  deficiencies  in  user's 
environment/  establish  new  goals /  determine  whether  it  is 
feasible  to  automate  the  business/  and  suggest  some 
acceptable  scenarios  (Yourdon/  1988:54).  The  final  step  is 
to  prepare  a  charter  that  will  guide  the  project  through  the 
development  process.  The  analysis  activity  transforms  two 
major  inputs/  user  policy  and  project  charter/  into  a 
structured  specification  (Yourdon/  1988:54).  The  design 
activity  allocates  requirements  in  the  specification  to 
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hardware  and  software  component  that  will  make  up  the  final 
system.  The  activity  of  implementation  "includes  both 
coding  and  integration  of  modules  into  a  progressively  more 
complete  skeleton  of  the  ultimate  system"  (Yourdon, 

1988:54).  The  acceptance  test  generation  activity  takes  the 
specification  and  uses  it  to  define  "an  acceptable  system, 
from  a  user's  point  of  view"  (Yourdon,  1988:54).  From  the 
specification,  acceptance  test  cases  are  generated  to  verify 
the  system  being  built.  Yourdon' s  quality  assurance 
activity  takes  the  completed  system  and  tests  it  against  the 
acceptance  test  that  was  generated.  The  procedure 
description  activity  is  the  generating  of  a  user's  manual 
that  both  describes  the  system  and  instructs  the  user  how  to 
use  it.  The  database  conversion  activity  applies  only  when 
the  new  system  is  replacing  an  older  one  or  when  the  new 
system  requires  a  current  user  database.  This  activity 
covers  the  process  of  taking  the  current  user's  database  and 
making  it  part  of  the  new  system.  The  last  event  or 
activity  is  installation.  This  activity  can  range  from 
cutting  over  to  the  new  system,  to  installing  the  system, 
training  personnel,  and  supplying  complete  system 
documentation . 

Yourdon's  life  cycle  of  activities  appears  to  be 
complete.  If  applied  correctly,  it  could  supply  the  events 
needed  to  evaluate  the  contractor's  performance. 

In  Implementing  Software  EnoineerinQ  Practices  Buckley 
describes  a  software  life  cycle  using  five  milestones,  three 
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reviews,  and  two  test  steps  (Buckley,  1989:15).  The 
milestones  are  software  requirements  review,  preliminary 
design  review  (PDR),  and  critical  design  review  (CDR),  and 
two  types  of  tests  are  code  and  unit  test,  and  acceptance 
test  (Buckley,  1989:15). 

The  author  defines  software  requirements  review  as  the 
formal  documentation  and  review  for  the  requirements 
definition  phase  of  the  development  life  cycle.  Buckley 
believes  this  is  a  good  point  in  the  development  process  to 
give  the  customer  a  degree  of  assurance  that  work  is 
proceeding  properly  (Buckley,  1989:80).  Software 
requirements  review  is  also  the  time  period  when  the 
customer  reviews  and  approves  the  software  requirements 
specifications.  PDR  is  defined  by  Buckley  as  where  the  top- 
level  architecture  for  each  computer  software  configuration 
item  (CSCI)  is  reviewed  and  approved  by  the  customer.  This 
review  deals  with  the  software  architecture.  Integration, 
draft  test  procedures,  and  draft  user  documentation 
(Buckley,  1989:88).  CDR  reviews  the  software's  detailed 
design.  This  is  done  by  reviewing  the  detailed  design 
documentation  (Buckley,  1989:121).  Buckley  defines  code  and 
unit  testing  as  when  code  and  units  functions  are  verified. 
These  Informal  tests  are  to  verify  the  code  or  unit,  as 
designed,  will  work  and  pass  the  formal  tests  (Buckley, 
1989:128).  Acceptance  tests  allow  the  customer  to  evaluate 
the  software  system.  This  evaluation  determines  if  the 
customer  will  accept  the  system  (Buckley,  1989:128-129). 
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Buckley's  life  cycle  Is  a  good  starting  point  for 
developing  events  and  milestones  to  evaluate  the 
contractor's  performance.  The  design  reviews  are  very 
similar  to  the  ones  used  by  the  government,  but  are  not 
complete  and  his  approach  does  not  use  formal  audits. 

First  published  in  1983  and  revised  in  1986  and  1990, 
DMSC's  Systems  Enaineerina  Management  Guide  was  written 
before  the  DSMC  MCCR  Management  Guide  and  DoD-STD-2167A, 
which  used  many  of  its  terms  and  definitions.  The  guide 
divides  the  software  development  cycle  into  seven  reviews 
and  two  audits  system  requirements  review.  These  are  the 
seune  reviews  and  audits  as  DoO-STD-2167A. 

The  seven  reviews  and  two  audits  are  major  events  and 
milestones  that  can  be  used  to  evaluate  contractor 
performance . 

Whitten's  approach  differs  from  the  majority  of  the 
authors  reviewed.  In  his  book.  Managing  Software  Projects; 
Formula  for  Success,  he  presents  a  software  development 
cycle  that  contains  15  separate,  but  Interlinked, 
activities.  These  activities  are  objectives, 
specifications,  high-level  design,  publications 
specifications,  test  plans,  low-level  design,  code,  unit  and 
functional  test,  component  test,  publications  draft  one, 
system  test,  publications  draft  two,  regression  test, 
package,  and  final  delivery  (Whitten,  1990:12). 

The  objectives  activity  entails  developing  a  document 
that  "defines  the  requirements  and  operational  need  that 


20 


must  be  satisfied  for  a  new  or  enhanced  product"  (Whitten, 
1990:13).  This  document  provides  the  underlying  direction 
to  follow  as  product  functional  and  design  tradeoffs  are 
made  throughout  the  development  cycle  (Whitten,  1990:13). 

Whitten's  specifications  activity  develops  documents 
that  describe  In  detail  "the  externals  of  the  product" 
(Whitten,  1990:13).  These  documents  describe  what  the 
product  will  look  like  to  the  user  (Whitten,  1990:13). 

The  activity  of  high-level  design  Identifies  and 
documents  the  components  that  make  up  the  product,  the 
functional  mission  for  each  component,  the  Interface  across 
these  components,  and  the  external  Interface  to  the 
operating  environment  (Whitten,  1990:14). 

Publications  and  specifications  activity  covers  the 
content  and  layout  of  each  publication  to  be  delivered. 

This  activity  provides  the  user  publications  that  accompany 
the  product,  and  provides  Information  on-line  for  the  users 
of  the  product  (Whitten,  1990:15). 

The  test  plans  activity  develops  the  documents  "that 
describe  the  who,  what,  when,  where,  and  how  for  a 
designated  test"  (Whitten,  1990:15).  Test  plans  are  written 
for  each  activity.  These  Include,  for  example,  unit  test, 
functional  test,  and  system  test  (Whitten,  1990:15). 

Low-level  design  covers  two  separate  levels  of  design. 
The  first  level  deals  with  how  modules  technically  work  with 
each  other  (external  Interfaces),  while  the  second  level 
designs  how  the  module  works  Itself  (Whitten,  1990:15-16). 
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Whitten's  coding  activity  covers  the  actual  writing  of 
software  based  on  the  low-level  design  documents. 

For  the  unit  and  functional  test  activity,  the  author 
defines  unit  testing  as  the  first  time  the  code  Is  run  or 
executed  (Whitten,  1990:16).  This  Is  different  for 
functional  testing,  where  two  or  more  modules  are  tested 
together  (Whitten,  1990:16). 

The  component  test  activity  Is  different  from  unit  and 
functional  testing,  because  this  Is  where  some  or  all  the 
product  components  are  tested  together  (Whitten,  1990:17). 

The  publications  draft  one  activity  Is  the  first  time 
the  product's  publications  are  reviewed  by  groups  within  the 
project  (Whitten,  1990:17).  The  author  says  these  documents 
are  the  user  documentation  that,  when  finalized,  will  be 
delivered  with  the  product. 

The  system  test  activity  can  be  an  Independent  or 
formal  test  and  can  be  performed  by  programmers  who  did  not 
develop  the  code.  If  desired.  It  can  Include  test  subjects 
that  represent  users  of  the  product  (Whitten,  1990:18). 

Publication  draft  two  activity  starts  when  the  comments 
from  draft  one  are  received  and  ends  when  the  final  document 
Is  printed  (Whitten,  1990:19). 

The  regression  test  activity  starts  after  the  system 
test  Is  complete  and  Involves  running  a  selected  set  of  test 
cases  against  the  software  and  supporting  hardware  (Whitten, 
1990:19).  Test  cases  are  rerun  If  any  problems  are  found 
and  fixed  during  this  activity. 
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The  packaging  activity  "Involves  collecting  the  pieces 
of  the  product  for  delivery  to  the  customer"  (Whitten, 
1990:19).  This  Includes  all  software  code,  hardware,  and 
user  publications. 

The  last  and  final  activity  Is  dellveiry.  This  Is  the 
point  where  the  product  Is  delivered  to  the  user  (Whitten, 
1990:15) . 

These  15  activities  provide  a  good  outline  of  the 
events  or  milestones  needed  to  evaluate  contractor 
performance,  with  most  applying  to  government  software 
acquisition. 

In,  Managing  the  Software  Process.  Humphrey  uses  a 
concept  of  the  project  plan  (Humphrey,  1990:86).  This  plan 
contains  five  steps.  These  steps  are:  goals  and 
objectives,  work  breakdown  structure  (WBS),  product  size 
estimates,  resource  estimates,  and  project  schedule.  The 
goals  and  objectives  step  "describes  what  Is  to  be  done,  for 
whom,  and  by  when,  as  well  as  the  criteria  for  determining 
project  success"  (Humphrey,  1990:87).  A  WBS  "subdivides  the 
project  Into  tasks  that  are  each  defined,  estimated,  and 
tracked"  (Humphrey,  1990:87).  The  product  size  estimates 
are  "quantitative  assessments  of  the  code  required  for  each 
product  element  (subsystem,  component,  or  module)" 

(Humphrey,  1990:87).  Resource  estimates  are  "based  on  prior 
experience,  known  productivity  factors  are  applied  to  yield 
reasonable  estimates  of  the  resources  required  for  each  WBS 
element"  (Humphrey,  1990:87).  The  basis  for  project 
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schedule  is  "the  available  project  staffing  and  resource 
estimates,  a  schedule  for  the  key  tasks  and  deliverable 
Items"  (Humphrey,  1990:87). 

This  project  plan  becomes  the  guiding  document  for  the 
software  development  process.  The  author  believes  "The 
project  plan  provides  a  definition  of  each  major  task,  an 
estimate  of  the  time  and  resources  required,  and  a  freunework 
for  management  review  and  control  (Humphrey,  1990:110).  The 
major  tasks,  time  and  resources  within  the  project  plan  are 
good  candidates  for  evaluating  contractor  performance. 

Youll's  book.  Making  Software  Development  Visible: 
Effective  Project  Control,  does  not  cover  major  events  or 
milestones.  Instead,  his  book  outlines  the  tools  needed  to 
gain  visibility  of  what  Is  happening  during  the  software 
development  process.  In  Youll's  chapter  on  process 
visibility,  he  says  "Visibility  of  the  effects  of  the 
process  will  provide  evidence  of  the  effectiveness  of  a 
particular  process  or  of  a  change  to  the  process"  (Youll, 
1990:109).  The  author  feels  that  process  visibility  will  be 
gained  by  understanding  the  process  and  monitoring  resource 
utilization. 

To  understand  the  process,  Youll  looks  at  three  areas: 
rework,  cost  of  change,  and  why  the  change  occurred.  Rework 
occurs  when  software  does  not  perform  within  required 
specification,  and  can  be  caused  by  mistakes  or  by  the 
customer  changing  requirements.  Youll  states  the  process 
can  be  Improved  If  three  questions  can  be  answered:  One, 
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what  types  of  defects  are  occurring?  Two,  where  are  the 
defects  occurring?  Three,  can  the  defects  be  detected 
earlier?  (Youll,  1990:114).  Tracking  the  cost  of  change 
through  the  development  process  allows  Insight  Into  the  cost 
of  defect  removal  as  well  as  tracking  cost  growth  In  system 
development.  Identifying  the  number  and  cause  of  change 
allows  for  proper  attention  to  be  applied  so  they  can  be 
decreased.  Youll  believes  all  root  changes  and  repercussion 
changes  need  to  be  monitored  (Youll,  1990:114).  Root 
changes  are  defined  as  changes  In  design  or  user 
requirements  that  cause  repercussion  changes.  In  the  area 
of  resource  monitor,  Youll  believes  tracking  resources 
allocation  and  development  progress  allows  visibility  of  the 
current  development  status  as  well  as  allows  for  the 
prediction  of  future  requirements  and  performance. 

Youll 's  approach  for  software  development  visibility 
can  be  tailored  to  evaluate  contractor  performance. 

The  software  development  process  described  In  DMSC's 
Mission  Critical  Computer  Resources  Management  Guide 
"presents  an  overview  of  the  activities  of  an  Integrated 
software  and  hardware  system  as  reflected  In  DoD-STD-2167A" 
(DSMC,  1991:5-2).  The  development  process  described  In  this 
guide  refers  to  the  same  nine  milestones  described  In  DoD- 
STD-2167A  and  DSMC '8  Systems  Enolneerlna  Management  Guide, 
but  calls  them  events  and  defines  when  each  event  Is 
accomplished  slightly  different  from  them.  SRR  "may  be  held 
after  the  Initial  determination  of  system  functions  and  the 
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preliminary  allocation  of  these  functions  to  configuration 
Items"  (CIS)  (DSMC,  1991:5-2).  This  review  also  provides  a 
look  at  the  "developer's  direction,  progress,  and 
convergence  on  a  system  configuration"  (DSMC,  1991:5-2). 

SDR  covers  "all  system  requirements  In  order  to  establish 
the  functional  baseline  documented  by  the  system 
specification"  and  allocates  hardware  Cl  and  computer 
software  Cl  (CSCI)  requirements  (DSMC,  1991:5-2).  SSR 
occurs  when  the  allocated  baseline  Is  established  for 
software.  The  software  preliminary  design  Is  reviewed  at 
PDR.  From  SSR  to  PDR  "the  developer  will  conduct  Informal 
design  reviews.  Inspections,  and  walkthroughs  to  evaluate 
the  progress  and  correctness  of  the  design  for  each  software 
component"  (DSMC,  1991:5-7).  These  Inspections  "serve  as 
the  basis  for  material  presented  at"  PDR  (DSMC,  1991:5-7). 
CDR  Is  the  culmination  of  design  activity  and  "should  assure 
that  software  design  satisfies  the  requirements  of  both  the 
system  level  specification  and  the  software  development 
specifications  (DSMC,  1991:5-9).  Next  Is  TRR,  which  Is  a 
formal  review  of  the  contractor's  readiness  to  begin  formal 
software  Cl  testing  and  done  after  both  Individual  modules 
and  components  are  Integrated  and  tested,  and  software  test 
procedures  are  reviewed  (DSMC,  1991:5-6).  PCA  "Is  the 
formal  technical  examination  of  the  as-built  software 
product  against  Its  design"  (DSMC,  1991:5-11).  FCA  Is  a 
verification  by  the  government  that  the  CSCI's  the 
contractor  developed  will  perform  In  accordance  with 


26 


requirements  and  Interface  specifications  (DSMC,  1991:5-11). 
This  is  done  by  examining  the  test  results  and  reviewing 
both  the  operational  and  support  documentation.  (DSMC, 
1991:5-11).  PCA  may  either  be  followed  by  or  concurrent 
with  FCA  (DSMC/  1991:5-11).  The  last  event  is  FQR,  which 
verifies  system  performance  complies  with  system 
requirements  (DSMC,  1991:5-12). 

The  nine  events  outlined  above  can  become  the  basis  for 
evaluating  contractor  performance.  They  are  all  major 
events  or  milestones  that  must  be  successfully  completed  if 
quality  software  is  to  be  delivered  at  the  scheduled  time 
and  at  the  agreed  price. 

Question  2  Summary.  These  articles  and  books  presented 
several  approaches  for  events  or  milestones  in  the  software 
development  process  that  could  be  used  to  evaluate 
contractor  performance.  This  research  found  a  strong 
consensus  on  major  events  or  milestones  in  the  software 
development  process. 

There  are  nine  common  events  or  milestones  within  each 
of  the  readings  that  can  be  used  to  evaluate  contractor 
performance.  Except  for  Humphrey,  all  the  authors  use  a 
combination  of  reviews,  audits,  or  tests  as  events  or 
milestones  in  the  software  development  process.  His 
approach  used  goals  and  objectives,  work  breakdown 
structure,  resource  estimates,  and  project  schedule  to 
evaluate  performance.  However,  Humphrey's  project  schedule 
did  contain  events  or  milestones  similar  to  the  other 
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authors.  The  consensus  of  milestones  that  should  be  used 


are  systems  requirements  review  (SRR),  system  design  review 
(SDR),  software  specification  review  (SSR),  preliminary 
design  review  (PDR),  critical  design  review  (CDR),  test 
readiness  review  (TRR),  functional  configuration  audit 
(FCA),  physical  configuration  audit  (PCA),  and  formal 
qualification  review  (FQR) .  These  specific  ncunes  were  not 
used  by  all  the  authors  to  describe  the  events  In  their 
software  development  life  cycle,  nor  do  all  the  phases  fit 
every  government  software  development  project.  Yourdon 
states  "certainly  every  project,  whether  structured  or  not, 
goes  through  some  kind  of  system  analysis,  design,  and 
Implementation"  (Yourdon,  1988:45).  These  nine  events  and 
milestones  should  be  tailored  for  each  software  development 
project. 

Question  3.  How  should  award  fee,  as  described  In  the  FAR 
and  Its  supplements,  and  software  development  events  be 
linked  to  Influence  contractor  performance? 

The  possible  linkages  between  award  fee  and  software 
development  events  appear  to  be  many.  Table  1  shows  four 
possible  contract  options.  The  types  of  contract  are  the 
key  FAR  management  tools  available  to  influence  contractor 
performance  during  the  software  development  process,  and 
award  fees  are  the  most  useful  In  influencing  contractor 
performance  throughout  the  software  development  process. 

The  benefit  of  award  fee  are  summarized  by  the  following: 
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The  award  fee  contract  provides  not  only  profit  or  fee 
motivation,  but  also  the  motivation  resulting  from 
periodic  evaluations  by  one's  professional  peers.  In 
addition,  it  offers  evaluation  flexibility,  in  two 
forms : 

(1)  the  flexibility  to  evaluate  on  a  judgmental  basis 
taking  into  consideration  both  contractor  performance 
levels  and  the  conditions  under  which  such  levels  were 
achieved;  and 

(ii)  the  flexibility  to  adjust  evaluation  plans  quickly 
to  reflect  changes  in  government  management  emphasis  or 
concern.  (NASA,  1989:1) 

Question  3  Summary.  By  combining  the  flexibility  of 
award  fee  with  the  events  outlined  in  Question  2,  the 
government  can  possess  a  workable  means  to  influence  the 
software  development  process  and  provide  the  contractor  with 
feedback.  An  example  of  this  is  Appendix  A,  Performance 
Evaluation  Periods,  where  the  award  fee  periods  are  tied  to 
events  using  a  fee  percent  allocation. 


Question  4 .  Can  the  award  fee  process  be  used  to  Influence 
contractor  performance  during  software  development? 

The  literature  reviewed  did  not  give  a  clear 
perspective  on  this  question.  To  answer  this  question, 
research  has  be  done  using  the  perspectives  of  current 
program  directors. 


Question  5.  What  criteria  should  be  used  to  evaluate 
contractor  performance? 

The  criteria  for  evaluating  contractor  performance 
during  the  evaluation  period  could  range  from  identification 
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of  development  problem  causes  and  solutions  to  government 
evaluation  of  contractor  developed  dociimentatlon .  The 
criteria  should  not,  however,  be  fragmented  over  a  large 
number  of  performance  areas  and  factors  because  It  dilutes 
the  award  fee's  emphasis  and  effectiveness  (NASA,  1989:4; 
AFSC ,  undated : 2  2 ) . 

Instead,  broad  performance  areas  should  be  selected, 
such  as  technical  and  business  management,  supplemented 
by  a  limited  number  of  subfactors  describing 
significant  evaluation  elements  over  which  the 
contractor  has  effective  management  control.  (NASA, 
1989:4) 

As  the  contractor  progresses  through  the  software 
development  process,  the  award  fee  criteria  should  change. 
The  award  fee  process  allows  the  criteria  to  be  changed 
before  the  start  of  the  next  evaluation  period  as  long  as 
the  contractor  receives  the  new  criteria  before  the  start  of 
the  next  period  (NASA,  1989:4). 

Question  5  Summary.  The  exact  criteria  Is  dependent  on 
the  specific  software  development  program.  The  research  has 
used  the  perspectives  of  current  and  former  program 
directors  to  find  the  current  system  program  office 
philosophy  on  criteria. 

Question  6.  What  type  and  form  of  feedback  should  be  given 
to  the  contractor  to  Improve  his  ability  to  develop 
maintainable  software  on  time,  and  within  cost? 

The  feedback  to  the  contractor  can  come  In  the  form  of  a 
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written  or  oral  presentation  by  the  government  at  the  end  of 
each  award  fee  period.  The  written  or  oral  presentation 
could  review  strong  and  weak  points  used  In  evaluation  and 
major  changes  required  by  the  contractor  In  the  next  period 
to  Improve  his  rating  (NASA,  1989:32-36). 

Question  6  Summary.  The  literature  does  not  fully 
answer  this  question.  So,  the  researcher  has  made  an 
attempt  to  compile  data  that  will  augment  the  available 
literature. 

Summary 

The  literature  review  provided  many  possible  solutions 
to  the  questions  In  Chapter  I.  The  solutions  discussed 
above  present  one  possible  solution  set.  They  may  not  be 
the  optimal  solution  set,  because  each  software  development 
project  Is  different.  These  differences  may  appear  to  be 
small  on  the  surface,  but  most  are  like  viewing  a  floating 
Iceberg.  You  know  there  are  large  differences  that  are 
currently  unseen.  A  contract  with  award  fee  periods  and  the 
award  fee  criteria  In  Appendix  A,  Performance  Evaluation 
Periods .  and  Appendix  B,  Evaluation  Criteria,  are 
recommended  starting  points. 

Chapter  III  covers  the  methodology  used  to  augment  the 
answers  to  research  questions  two  through  six.  This  was 
done  by  surveying  Program  Managers,  Deputy  Program  Managers, 
Contracting  Officers,  and  contractors.  The  potential 
population  for  this  sample  Is  very  large  and  contains  people 
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both  with  and  without  award  fee  experience.  The  sample 
frame  chosen  for  this  research  Included  personnel  with  award 
fee  experience. 
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Ill .  Methodology 


Introduction 

A  literature  search  and  survey  were  the  method  used  for 
the  research.  The  literature  search  reviewed  articles 
written  In  professional  journals,  text  books.  Government 
publications,  plus  specifications  and  standards  addressing 
the  software  development  process  and  award  fees.  The  survey 
focused  on  Progreun  Managers,  Deputy  Program  Managers, 
Contracting  officers,  and  contractors  who  have  experience 
with  award  fee  In  general,  and  specifically  with  award  fee 
In  software  acquisition.  A  judgment  sample  was  used  for  two 
reasons:  (1)  the  opinions  of  personnel  experienced  with 

award  fee  were  used  to  gain  Insight  Into  current  policies, 
experience,  and  effectiveness  of  award  fees  and  (2)  this 
experience  base  was  used  to  develop  a  draft  list  of  award 
fee  criteria.  (Emory  and  Cooper,  1991:275-277) 

Methodology 

There  are  different  ways  of  surveying  a  population, 
including  mailed  questionnaires,  telephone  Interviews,  and 
personal  Intezrvlews.  Research  has  shown  that  telephone  and 
personal  Interviews  provide  more  reliable  data  then  mailed 
questionnaires  and  avoid  potential  mlslnterpretlon  of 
questions  (Emory  and  Cooper,  1991:338-339;  Williams,  1992). 
This  research  primarily  used  personal  and  telephone 
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Interviews,  and  only  resorted  to  mailed  questionnaires  when 
the  Interviewee  was  unavailable  for  a  personal  Interview. 

Justification 

The  survey  development  process  focused  on  developing 
questions  that  helped  develop  Insight  Into  the  current 
status  of  award  fee  experience  and  use  In  the  program  office 
as  well  as  the  effect  award  fee  Is  having  or  would  have  on 
software  development.  Questions  will  be  developed  using  the 
design  of  experiment  approach  (McClave  and  Benson,  1991:859- 
865).  The  research  analysis  combines  statistical  and 
judgmental  analysis.  The  design  of  experiment  approach 
focuses  on  developing  questions  that  facilitate  statistical 
analysis . 

Method  Choice 

Personnel  Interviews  were  conducted  at  Aeronautical 
Systems  Division  with  Program  Managers  and  Deputy  Program 
Managers  and  with  contractors  in  the  Dayton,  Ohio  area  who 
belong  to  the  National  Security  Industrial  Association 
(NSIA) .  Telephone  Interviews  were  used  for  managers  at 
Electronic  Systems  Division  and  Space  Systems  Division,  and 
for  contractors  affiliated  with  NSIA  who  were  not  located  In 
the  Dayton,  Ohio  area.  This  is  a  representative  sample  of 
the  Air  Force  personnel  experienced  with  award  fee 
contracts.  To  feunillarize  the  interviewees  with  the  pending 
interview,  a  copy  of  an  introductory  cover  letter  (Appendix 
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C  and  E)  and  Interview  guide  (Appendix  D  and  F)  was  sent  to 
each  interviewee  before  the  actual  interview  took  place. 

Research  Process 

The  research  process  was  divided  into  five  specific 
parts:  reviewing  existing  literature,  preparing  the 
interview  guide,  sample  selection,  conducting  interviews, 
and  analyzing  the  data. 

Literature  Review.  An  extensive  literature  review  was 
conducted  using  data  primarily  from  the  last  five  years  that 
focuses  on  the  technical  aspects  of  software  development  and 
the  award  fee  process.  The  literature  review  is  contained 
in  Chapter  II. 

Interview  Guide.  Interview  questions  were  developed 
with  two  different  populations  in  mind.  Government  and 
contractor.  The  interview  questions  address  three  specific 
areas:  "Program  Director  Information,”  "Program 

Background,"  and  "Award  Fee  Perspective.” 

The  "Program  Director  Information”  area  gathered  data 
on  the  interviewee's  position,  both  current  and  past,  and 
his  general  acquisition  experience. 

In  the  area  of  "Program  Background",  data  was  gathered 
on  the  Interviewee's  award  fee  experience. 
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The  last  area,  "Award  Fee  Perspective,"  examined  the 
Interviewee's  personal  .views  on  how  award  fees  should  be 
used  In  software  acquisition. 

The  question  development  process  included  numerous 
iterations  to  Insure  clarity  and  validity.  The  review 
process  started  with  a  review  of  the  proposed  questions  by 
Instructors  and  past  Air  Force  Institute  of  Technology 
students  familiar  with  surveys.  The  interview  guide  was 
then  forwarded  to  a  program  office  and  commercial  contractor 
with  existing  software  award  fee  contracts  for  validation. 
Their  selection  was  based  on  the  researcher's  personal 
experience  with  both. 

Sample  Selection.  Sample  selection  was  done  by 
contacting  the  Contracting  Policy  Offices  at  Aeronautical 
Systems  Division,  Electronic  Systems  Division,  and  Space 
Systems  Division  to  determine  which  Progreun  Directors  had 
experience  with  award  fee  contracts.  This  process  also 
allowed  access  to  personnel  who  had  transferred  from  an 
award  fee  program  to  a  non  award  fee  program.  Next,  the 
contracting  directorate  of  each  progreun  with  an  award  fee 
was  contacted  to  obtain  a  copy  of  the  award  fee  plan. 

Interview  Process.  Each  Government  Interviewee 
received  a  cover  letter  (Appendix  C)  from  the  Aeronautical 
Systems  Division  (ASD)  Vice  Commander  and  a  copy  of  the 
Government  Interview  Guide  (Appendix  D) .  The  cover  letter 
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explained  the  purpose  of  the  interview,  defined  key  teznns, 
and  discussed  the  attached  material.  In  addition,  the 
letter  requested  that  an  interview  date  and  time  be 
established.  The  contractor  interviewee  received  a  cover 
letter  similar  to  the  ASD  Vice  Commander  from  the  researcher 
(Appendix  E)  and  a  copy  of  the  Contractor  Interview  Guide 
(Appendix  F) . 

Analysis  of  the  Data.  The  survey  results  were  used  to 
establish  an  experience  base  from  which  some  of  the  research 
questions  can  be  answered.  Data  from  the  "Program  Director 
Information"  interview  section  was  used  to  identify  the 
interviewee's  experience  in  program  offices  and  what 
specific  areas  within  the  program  office.  The  "Program 
Background"  area  supplied  data  concerning  award  fee  and 
software  experience.  The  last  interview  guide  section  deals 
with  interviewee  options  as  to  the  effectiveness  of  award 
fee  on  the  software  development  process.  Each  of  these 
sections  contain  questions  that  can  be  analyzed  by  either 
statistical  or  judgmental  means. 

Statistical  analysis  of  questions  included  histograms, 
bar  charts  and  data  plots  to  find  trends  or  locus  of 
consensus.  The  judgmental  analysis  looked  at  the  range  of 
responses  for  each  question  and  then  tried  to  find  a 
consensus  of  answers.  These  results  will  be  opinions  of 
managers  with  experience  with  award  fee  contracts  and  are 
useful  as  lessons  learned  for  future  award  fee  contracts. 
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The  sample  size  from  Government  personnel  was  twenty- 
three  and  from  contractor  personnel  was  only  four.  .The 
Government  seunple  size  Is  large  enough  to  make  Inferences 
about  the  population  of  programs  with  award  fee.  The 
contractor  sample  size  Is  small  and  does  not  allow  the 
researcher  to  make  any  authoritative  Inference  about 
contractor  views  of  award  fees  (Emory  and  Cooper,  1991:259- 
264).  The  contractor  Interviews  where  used  to  see  If  any 
large  differences  In  view  point  existed. 

The  researcher  found  It  very  difficult  to  coordinate 
telephone  Interviews  with  Program  Directors  at  both  Space 
Systems  Division  and  Electronic  Systems  Division.  During 
the  development  of  the  questionnaire,  the  researcher  found 
It  very  difficult  to  Identify  contractors  with  award  fee 
contracts.  The  original  plan  was  to  have  the  ASD  Vice 
Commander's  cover  letter  Include  a  request  for  the  program 
director  to  forward  a  copy  of  the  contractors  survey  to 
their  contractor.  The  ASD  Deputy  Chief  of  Staff  for 
Contracting  did  not  concur  with  this  approach  because  of  a 
concern  that  the  contractor  may  charge  the  Government  for 
the  assistance  (Appendix  G) .  The  four  contractors  surveyed 
agreed  to  respond  to  the  survey  at  no  cost  to  the 
Government . 

Summarv 

This  chapter  discussed  the  methodology  research 
approach  used  for  this  thesis.  The  research  approach 
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Included  a  literature  search  and  survey.  The  literature 
search  reviewed  current  text  books  and  articles  covering 
software  development  and  award  fees.  Air  Force  military  and 
civilian  and  contractor  personnel  feunlllar  with  award  fee 
contracts  were  surveyed  by  personal  or  telephone  Interview 
to  answer  to  research  questions. 

Chapter  IV  discusses  the  analysis  process  used  on  the 
Interview  data.  The  discussion  Includes  findings  with 
respect  to  each  question  and  factual  Information  from  data 
Interpretation,  Inference,  and  evaluation. 
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IV.  Findings 


Introduction 

The  researcher  conducted  27  Interviews  with  program 
directors,  deputy  program  directors,  program  managers, 
contracting  officers,  and  contractors,  between  10  January 
and  16  April  1992.  During  three  interviews,  both  the  Deputy 
Program  Director  and  Program  Director  for  the  System  Program 
Office  participated.  Nineteen  of  the  Interviewees  were 
military,  four  were  members  of  the  civil  sezrvlce,  and  four 
were  contractors. 

All  of  the  comments  from  both  government  and  contractor 
Interviewees  were  very  enlightening,  and  revealed  several 
facts.  First,  the  Interviewees  tended  to  have  the  same 
viewpoint,  l.e.,  award  fee  is  not  a  hindrance  to  the 
program.  Second,  nineteen  of  the  government  and  all  four  of 
the  contractor  Interviewees  plan  to  use  award  fees  on  future 
contracts.  Third,  sixteen  of  the  interviewees  feel  award 
fee  Is  very  useful  In  software  development  because  It 
focuses  both  government  and  contractor  attention  and  fosters 
communication . 

The  remainder  of  this  chapter  will  synthesize  the  data 
from  the  literature  search  and  Interviews.  First,  the 
sample  size  will  be  discussed  along  with  the  quantity  of 
responses  from  each  product  center.  Second,  the  Interviewee 
responses  to  each  question  will  be  combined  to  find  a  common 
position.  Last,  these  combined  responses  and  the  data  from 
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the  literature  search  will  be  used  to  address  the  research 


questions . 

Sample  Data 

The  government  Interview  sample  was  obtained  from  the 
Contracting  Policy  Offices  at  Aeronautical  Systems  Center 
(ASC)/  Wright  Laboratories  (WL),  Electronic  Systems  Center 
(ESC),  and  Space  and  Missiles  Systems  Center  (SSC).  This 
allowed  the  researcher  to  determine  which  Program  Directors 
had  experience  with  award  fee  contracts.  This  process  also 
allowed  access  to  personnel  who  had  transferred  from  an 
award  fee  program  to  a  non  award  fee  program.  The  number  of 
responses  from  ESC  and  SSC  were  lower  then  the  original 
research  had  planned.  This  was  due  to  the  Inability  of  the 
researcher  to  coordinate  Interview  times  with  Directors  not 
located  at  Wrlght-Patterson  AFB  Ohio.  This  research  also 
called  for  Interviewing  local  members  of  National  Security 
Industrial  Association  (NSIA) .  The  data  seunple  did  not 
Include  NSIA  member  data  because  the  researcher  did  not  have 
direct  access  to  members  and  was  not  able  to  follow-up  with 
the  NSIA  point  of  contact.  The  data  from  the  four 
contractors  (CONTR)  that  was  Included,  was  a  direct  result 
of  sponsorship  of  the  applicable  program  offices  Involved. 
Figure  5  shows  the  responses  received  based  on  the 
Interviewee's  location.  The  number  of  possible  responses 
(40)  for  the  government  Interviewees  was  derived  for  the 
Information  provided  by  the  respective  contracting  offices. 
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Number  of  Responses  Possible  Responses 


Figure  5.  Interviewee  Responses  by  Location 


Interview  Data 

The  following  section  will  summarize  the  Interviewee 
responses  for  each  question,  divided  by  government  and 
contractor  responses.  In  the  following  graphs,  the 
Interviewee  numbers  from  one  (1)  to  twenty  three  (23) 
represent  the  goverrunent  Interviewees,  while  numbers  twenty 
four  (24)  to  twenty  seven  (27)  represent  the  contractor 
Interviewees.  The  researcher  used  "N/C"  when  the 
Interviewee  did  not  comment  on  this  question. 


Program  Director  Information.  The  Interview  guide 
first  asked  for  the  date  the  Interviewee  was  assigned  to 
his/her  current  position.  Figure  6  shows  the  time.  In 


42 


Time  in  Current  Job 

Interview  Question  1  a 


Interviewee 


Figure  6.  Government  Time  in  Current  Position 

years,  each  government  Interviewee  has  spent  In  his  or  her 
current  assignment.  The  years  range  from  a  high  of  6.1  to  a 
low  of  0.7  years,  with  an  average  of  2.4  years.  Contractor 
Interviewees  had  a  higher  average  time  In  current  posltlon- 
3.5  years.  Figure  7  shows  the  number  of  years  for  each 
contractor . 

The  second  part  of  question  1  asked  what  level  of 
certification  the  Interviewee  had  In  the  Acquisition 
Professional  Development  Program  (APDP),  Program  Management 
field.  Figure  8  shows  that  most  of  the  Interviewees  are 
certified.  The  figure  uses  an  asterisk  (*)  to  show  an 
Interviewee  was  certified  In  more  than  one  field.  The  "R" 

Is  used  to  show  a  request  for  level  three  certification  Is 
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Time  in  Years 


Figure  7.  Contractor  Interviewee  Time  in  Current  Position 


Program  Management  Level 

Interview  Question  1  b 


1  2  3  *  5  8  7  8  9  10  1 1  12  13  14  15  18  17  18  19  20  21  22  23 

Interviewee 


Figure  8.  Government  Interviewee  Program  Management 
Certification  Level 
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pending.  The  "*2"  is  used  to  show  the  interviewee  is 
certified  level  two  (2)  in  the  APDP  Contracting  field. 

Question  2  deals  with  the  government  and  contractor 
interviewees  prior  experience  as  a  director  or  deputy 
program  director  and  what  specific  programs.  Of  the 
individuals  interviewed/  40  percent  have  prior  experience 
(Figure  9).  The  second  part  of  this  question  was  designed 
to  gain  insight  into  the  type  of  experience  the  interviewee 
had.  The  government  and  contractor  interviewees  with  prior 
experience  were  mostly  with  large  programs  like  the  B-1 
Bomber  Program  or  a  group  of  smaller  basket  programs  under 


Prior  Director  or  Deputy 

Question  2 


16 


Response 

Figure  9.  Government  and  Contractor  Interviewee  Experience 
as  Director  or  Deputy  Program  Director 
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one  director/  for  example/  the  Air-to-Surface  Guided 
Missiles  program  office. 

To  understand  the  experience  base  of  both  goverrment 
and  contractor  interviewees/  question  3  asked  for  the 
length  of  time  the  interviewee  had  been  in  a  System  Progr£un 
Office  (SPO).  The  responses  ranged  for  zero  (0)  to  thirty 
(30)  years  with  an  average  of  10.4  years  (Figure  10). 
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Figure  10.  Government  and  Contractor  System  Program 
Office  Time 


The  government  interviewees  were  asked  for  their  Air 
Force  Specialty  Code  (AFSC).  When  the  four  government 
civilians  where  asked  for  their  AFSC/  the  researcher  asked 
that  they  be  specified  in  terms  of  a  military  AFSC.  Eleven 
of  the  twenty  three  government  interviewees  had  more  than 
one  AFSC.  The  0029  AFSC  designates  a  Program  Director  while 
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an  AFSC  of  2916  designates  a  Program  Manager,  the  2716  Is  an 
Acquisition  Specialist,  and  a  2816  AFSC  is  an  Engineering 
Specialist.  Figure  11  shows  the  distribution  of 
interviewees  in  each  AFSC. 


Interviewee  AFSC 

Interview  Question  4 


0029  2716  2806  2916  N/C 


Air  Force  Specialty  Code  (AFSC) 

Figure  11.  Government  Interviewee  Air  Force  Specialty  Codes 
(AFSC) 

Question  5  looked  at  what  areas  within  the  program 
office  the  Interviewees  have  worked.  Eleven  of  the  twenty- 
three  government  interviewees  had  experience  in  more  than 
one  area.  Most  of  the  Interviewees  had  either  experience  in 
engineering  or  program  management/projects  (20  out  of  27), 
while  the  remaining  seven  have  experience  in  either 
manufacturing,  contracting,  integration,  or  configuration 
management . 


Program  Background.  This  portion  of  the  interview 
guide  focused  on  uncovering  the  award  fee  experience  the 
interviewees .  Question  6  looked  at  the  beginning  of  the 
award  fee  process.  It  asked  if  the  interviewee  had  taken 
part  in  the  precontract  award  fee  process  and,  if  they  had, 
on  what  progr2tm  or  programs.  Sixty-nine  percent  of  the 
interviewees  that  answered  the  question,  both  government  and 
contractor,  had  taken  part  in  the  precontract  award  fee 
process  (Figure  12).  The  programs  cited  in  part  two  of  this 
question  are  large  and  again  cover  a  wide  range  from 
Aircraft  Engines  to  Vertical  Launch  Systems. 


Prior  Experience  with  Award  Fee 

Question  6 


_ Response _ 

Figure  12.  Government  and  Contractor  Experience  with 
Precontract  Award  Fee 
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The  seventh  question  asked:  Does  your  program  office 
currently  have  an  award  fee  contract?  This  question 
requested  Information  on  the  number  of  award  fee  contracts 
In  existence  at  the  time  on  this  research.  The  Individuals 
Interviewed  were  selected  by  their  experience  or  their 
current  position  In  a  program  with  award  fee  contracts. 
Eighty-eight  percent  of  those  that  answered  the  question 
currently  had  award  fee  contracts.  The  N/A  In  Figure  13  was 
the  response  from  a  Fee  Determining  Official  (FDO)  the 
researcher  Interviewed.  All  four  of  the  contractors 
surveyed  had  award  fee  contracts. 


Award  Fee  Contracts 
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Questions  8  through  12  were  focused  on  the  Interviewees 
current  progr€un.  The  eighth  question  In  the  Interview  guide 
focused  on  what  portion  of  the  progr2un  the  award  fee 
covered.  This  allowed  the  researcher  to  see  where  award 
fees  are  currently  being  applied.  The  responses  ranged  from 
the  "Whole  Program"  to  "Qualification  of  Second  Source". 
There  was  no  common  application  of  award  fee,  and  the 
application  appeared  to  depend  directly  on  the  type  of 
program/ system  under  development.  The  researcher  also  found 
that  the  program's  development  phase,  question  nine,  also 
Influenced  the  application  of  award  fee,  question  eight. 

Question  9  asked  the  Interviewee  for  the  development 
phase  of  their  respective  programs.  The  phases  ranged  from 
concept  exploration  to  production.  Figure  14  shows  the 
distribution  of  programs  by  phase  and  uses  the  following 
abbreviations:  Concept  Exploration  (C/E),  Demonstration 
/Validation  (D/V) ,  Engineering  and  Manufacturing  Development 
(END),  Production  and  Deployment  (P/D),  and  Not  Applicable 
(N/A) .  Three  Interviewees  did  not  currently  have  award  fee 
contracts.  The  award  fee  total  In  Figure  14  equals  29 
because  one  Interviewee  had  three  award  fee  contracts;  one 
each  for  C/E,  D/V,  and  EMD. 

The  tenth  question  dealt  with  how  the  program's 
software  was  covered  In  the  award  fee  plan.  Most  of  the 
interviewees  said  that  software  development  was  covered 
either  as  a  specific  Item  or  with  "overall  program 
management."  Of  the  27  Interviewees,  the  FDO  did  not 
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Award  Fee  Programs  By  Development  Phase 

Question  9 


C/E  D/V  EMO  P/0  N/A 

_ Development  Phase _ 

Flgur«  14.  Award  Fee  Programs  By  Development  Phase 


comment  because  his  position  did  not  have  individual  program 
responsibility,  and  one  interviewee  did  not  responded  to  the 
question.  The  researcher  found  the  reason  that  some 
interviewees  answered  this  question  "no”  was  that  their 
specific  progreuns  were  mostly  hardware.  For  example,  it  was 
an  aircraft  engine  progreun  or  a  new  25mm  fuse  for  the  AC- 
130.  Nineteen  interviewees  said  that  the  award  fee  plan 
covered  software  development  while  five  out  of  27  said  it 
did  not. 

The  contract  values,  question  11,  were  of  interest  to 
see  if  there  was  any  correlation  between  the  contract  value 


and  the  award  fee.  Figure  15  is  a  modified  frequency 
distribution  of  contract  values.  The  contract  values  ranged 
from  $1  million  to  $12  billion.  The  frequency  bins  are  the 
value  shown  and  lower  until  the  next  lower  value  (i.e.,  1  = 
values  between  0 . 0  and  1.0,  5  *  values  between  1 . 1  and  5.0). 


Frequency  Distribution 
of  Contract  Dollar  Value 


Contract  Value  ($10  Million) 


Figure  15.  Frequency  Distribution  of  Contract  Dollar  Value 

Question  12  dealt  with  the  percent  award  fee  the 
contractor  had  received.  This  question  was  looking  for  a 
trend  in  award  fee  pool  percentage  that  contractors 
received.  The  data  shows  no  real  trend,  and  percentages 
ranged  from  30  to  100  percent.  Two  interviewees  supplied 
award  fee  percentages  by  period.  One  showed  a  progressive 
increase  (period  1>44%,  period  2»54%,  and  period  3^100%). 
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The  other  Interviewee's  data  varied  between  90  percent  and 
82  percent (period  1=90%,  period  2^82%/  period  3=83%,  and 
period  4=87%).  It  would  be  Interesting  to  see  If  the 
contractor  In  ex^unple  one  could  maintain  this  level  of 
performance  for  award  fee  period  four. 

Questions  13  and  14  dealt  with  the  Interviewees' 
experience  with  award  fee  contracts  and  software  development 
respectively.  These  two  questions  were  used  to  gain  some 
Indication  of  the  experience  level  of  the  Interviewees  and 
were  used  to  evaluate  the  Interviewee  responses  to  the 
questions  In  the  award  fee  perspective  portion  of  the 
Interview  guide.  All  the  Interviewees  had  at  least  some 
experience  with  award  fee  contracts.  Figure  16  shows  the 
distribution  of  Interviewee  experience.  The  majority  of 
Interviewees  had  considerable  experience  with  award  fee 
contracts.  Interviewee  software  development  experience  was 
less.  The  majority  of  Interviewees  had  some  software 
experience  (Figure  17).  Two  Interviewees  had  no  experience 
with  software  development.  This  may  have  been  due  to  the 
systems  launch  vehicle  they  were  developing.  These  specific 
systems  are  hardware- Intensive  and  have  only  a  minimal 
amount  of  software. 

Award  Fee  Perspective.  The  last  section  of  the 
Interview  guide  was  Award  Fee  Perspective.  The  researcher 
used  this  section  to  collect  data  for  the  Investigative 
questions  associated  with  this  research.  The  specific 
questions  were:  Question  4.  Can  the  award  fee  process  be 
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Number  Interviewees 
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Figure  16.  Award  Pee  Experience 


Figure  17.  Software  Development  Experience 
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used  to  influence  contractor  performance  during  software 
development?  Question  5 .  What  criteria  should  be  used  to 
evaluate  contractor  performance?  Question  6.  What  type  and 
form  of  feedback  should  be  given  to  the  contractor  to 
improve  his  ability  to  develop  maintainable  software  on 
time,  and  within  cost? 

Question  15  asked:  What  written  guidance  would  help  in 
the  development  of  award  fee  contracts?  The  interviewees 
from  Electronic  Systems  Center  (ESC)  referenced  ESC 
Commander's  Policy  letters  on  Award  Fee  Determination  and 
HQ  ESC/PK  Policy  letter  on  Award  Fee  Process  Excellence. 
Nineteen  of  the  twenty-four  government  interviewees  said 
that  examples  would  be  of  great  benefit.  These  examples 
should  include  types  of  contracts  that  award  fee  has  been 
used  with,  examples  of  both  good  and  bad  award  fee  plans, 
and  criteria,  and  objectives  most  appropriate,  to  name  a 
few.  The  interviewees  also  cited  center  policy  letters,  the 
Systems  Command  Award  Fee  Handbook,  pros  and  cons  of  award 
fee,  and  education  as  additional  areas  that  would  assist  the 
program  director  with  award  fee  contracts.  All  four 
contractors  interviewed  looked  for  clear  and  definitive 
instructions  on  how  the  award  fee  is  to  be  used.  They 
looked  for  criteria  that  are  "end  items,"  commitment  by  both 
parties  to  implement  the  plan,  and  clear  understanding  of 
requirements  to  name  a  few. 

Questions  16  through  23  asked  the  interviewees  their 
options  on  the  effects  award  fee  has  had  on  software 
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development.  The  questions  address  software  development, 

cost  control,  delivery  timeliness,  maintenance  costs, 

reliability,  documentation,  testability,  and  quality. 

These  questions  were  developed  to  address  research  question 

4:  Can  the  award  fee  process  be  used  to  Influence 

contractor  performance  during  software  development? 

Seven  of  the  twenty-three  government  interviewees,  and 

one  of  the  four  contractor  interviewees  had  similar  comments 

for  questions  16  through  23.  One  of  the  contractors 

interviewed  was  an  aircraft  engine  manufacturer  with  little 

engine  software  in  his  product.  In  this  case,  award  fee 

associated  with  software  development  would  have  little 

effect  on  contractor  performance. 

The  sixteenth  interview  question  asked  if  award  fee 

would  benefit  software  development.  Comments  include: 

benefit  can  not  be  quantified;  there  is  no  difference 

between  hardware  and  software  developments  with  award  fee; 

some  help  in  all  areas;  definitely,  award  fee  is  a 

motivator;  award  fee  is  a  great  benefit;  and  benefits  can 

not  be  quantified.  All  except  one  government  interviewee 

felt  that  award  fee  would  be  a  positive  Influence  on 

software  development.  The  one  government  interviewee  felt 

award  fee  would  be  a  hindrance  because: 

The  contractor  becomes  focused  on  agreed  upon  software 
requirements.  If  award  fee  were  In  place,  the 
contractor  would  not  be  willing  to  tweak  mission 
requirements  as  needed  in  the  development  process. 
Requirements  change.  (Appendix  H,  page  141) 
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One  of  the  contractor  interviewees  believed  that  the  award 
fee  process  was  of  benefit  to  the  whole  program;  for  his 
approach  was  to  deliver  the  highest  quality  product  that  can 
be  produced.  The  award  fee  process  supplied  the  feedback 
and  interim  evaluation  necessary  to  insure  the  customer 
received  a  quality  product  (quality  being  defined  as  giving 
the  customer  what  he  wants).  The  question  found  software 
development  would  be  helped  by  using  award  fees.  This  was 
arrived  at  by  88  percent  of  the  interviewees  responding 
positively  to  award  fees  affect  on  software  development. 

Question  17  addressed  the  affect  award  fee  would  have 
on  software  development  cost  control.  The  government  and 
contractor  responses  were  divided  into  seven  categories  not 
applicable  (N/A)/  no  comment  (N/C)^  Depends/  Not  Sure,  No, 
YeS/  and  Same  as  16.  The  term  "Depends"  was  used  when 
respondents  qualified  their  answer  with/  for  example:  if 
resource  management  is  an  award  fee  criteria  or  depends  on 
contract  type.  The  term  "Same  as  16"  means  the  interviewee 
used  the  same  answer  as  that  in  Interview  question  16. 

Figure  18  shows  only  five  of  twenty-three  government  and  two 
of  four  contractor  interviewees  believed  award  fees  would 
have  a  positive  effect  on  software  development  cost. 

Example  of  "yes"  responses  were:  helped  focus  management 
attention  on  specific  areas,  and  contractor  managers  would 
become  more  Involved  in  regulating  hours.  Some  of  the  "no" 
responses  by  interviewees  were:  not  an  effective  tool,  and 
no,  because  of  documentation. 
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Will  Future  Award  Fee  Contracts 
Effect  Software  Developnnent  Costs  (17) 

lOn - 


I _ Response _ 

Figure  18.  Award  Fee  Effect  on  Software  Development 
Cost  Control 

Question  18  asked  If  award  fee  would  Improve  software 
delivery  timeliness.  There  was  no  clear  consensus  on  this 
question.  Only  four  of  twenty-seven  Interviewees  said  "yes" 
(two  government  and  two  contractor).  The  interviewees  that 
answered  "no"  felt  that  by  Incentlvlzlng  timeliness  the 
government  would  be  sacrificing  system  performance. 

Figure  19  shows  the  distribution  of  interviewee  responses. 
The  responses  in  the  "might  help"  category  said  that  award 
fee  criteria  focusing  on  timeliness  may  force  better  up 
front  planning,  it  depends  on  the  metric,  and  this  is  where 
award  fee  should  be  focused. 
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Will  Future  Award  Fee  Contracts  Effect 
Software  Delivery  Timeliness  (18) 
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Figure  19.  Award  Fee  Effect  on  Software  Delivery  Timeliness 


Question  19  asked  about  software  maintenance  costs  and 
the  effect  award  fee  would  have  on  them.  As  with  question 
eighteen,  there  was  no  clear  consensus  about  its  effect. 

The  one  "yes"  felt  that  maintenance  schedules  could  be 
conformed  to  award  fee  criteria.  The  "no,"  "might  help," 
and  "tough  to  measure"  responses  had  a  common  theme.  This 
theme  was  that  most  maintenance  costs  are  Incurred  years 
after  the  system  is  delivered  to  the  government  and  after 
the  development  contract  is  closed.  It  is  very  difficult  to 
Implement  an  award  fee  plan  under  these  circumstances. 

Figure  20  shows  the  distribution  of  interviewee  responses. 
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Will  Future  Award  Fee  Contracts  Effect 
Software  Maintenance  Costs  (19) 
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Figure  20.  Award  Fee  Effect  on  Software  Maintenance  Costs 


The  Interviewees  were  asked  In  question  20  If  award  fee 
would  affect  software  reliability.  There  was  an  even  split 
between  Interviewees  who  did  and  who  did  not  believe  award 
fee  would  effect  software  reliability  (Figure  21).  The 
Interviewees  who  felt  award  fees  would  affect  software 
reliability  said  that  once  the  software  is  operational, 
reliability  can  easily  be  measured.  The  Interviewees  that 
answered  "no"  felt  It  would  not  reduce  the  number  of 
failures  and  meaningful  failure  measurement  will  be 
completed  too  far  in  the  future. 
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Will  Future  Award  Fee  Contracts  Effect 
Software  Reliability  (20) 
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Figure  21.  Award  Fee  and  Software  Reliability 

Question  21  looked  at  award  fee  and  its  effect  on 
software  documentation.  Eleven  of  the  twenty-seven 
Interviewees  believed  award  fee  would  have  a  large  positive 
effect  on  software  documentation.  Award  fee  can  be  used  as 
leverage  to  insure  timeliness  and  quality.  Figure  22  shows 
a  bar  chart  of  the  interviewee  responses.  The  one 
interviewee  that  responded  "can  not  tell"  was  unsure  of  the 
question. 

Question  22  looked  at  the  affect  award  fee  would  have 
on  software  testability  (Figure  23).  Six  interviewees 
thought  that  award  fee  would  affect  testability.  The 


Figure  22.  Award  Fee  and  Software  Documentation 


Will  Future  Award  Fee  Contracts  Effect 
Software  Testability  (22) 
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Figure  23.  Award  Fee  and  Software  Testability 


interviewees  that  said  "yes,"  believed  award  fee  would  focus 
contractor  attention  of  formal  test  and  demonstration  of 
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software  requirements.  Four  thought  award  fee  would  have  no 
affect  at  all.  One  Interviewee  believes  the  culture  within 
the  contractor  is  the  driving  force  behind  software 
testability.  Two  interviewees  said  "maybe.”  Their 
responses  said  the  affect  award  fee  would  have  is  dependent 
on  the  criteria  used  to  measure  contractor  performance. 

The  twenty-third  question  asked  the  interviewees  if 
software  quality  would  be  affected  by  award  fee.  As  in 
question  twenty- two  above,  the  difference  between 
interviewee  "yes"  and  "no"  responses  is  only  two  (Figure 
24).  Interviewees  said  "yes,"  if  specific  criteria  is 
measurable  and  meaningful.  Interviewees  responding  "no" 


Will  Future  Award  Fee  Contracts  Effect 
Software  Quality  (23) 
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Figure  24.  Award  Fee  and  Software  Quality 

believe  there  is  a  problem  in  defining  quality.  Quality  is 
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difficult  to  measure.  If  quality  is  defined  as  the  software 
working  and  it  does  not,  the  software  quality  problem  can  be 
solved  by  using  contractual  procedures  other  than  award  fee. 
The  Interviewees  that  responded  "maybe"  either  gave  no  more 
information  for  their  response,  or  said  it  may  be  a  by¬ 
product  of  good  up  front  planning  by  both  the  government  and 
contractor  in  Implementing  an  award  fee  plan.  The  "can  not 
tell"  responses  focused  on  the  specific  definition  of 
software  quality. 

In  questions  16  through  23  several  main  points  were 
revealed.  First,  a  Computer  Software  Improvement  Progrcun 
(CSIP)  with  its  Software  Capability  Evaluation  (SCE)  can  be 
used  as  award  fee  evaluation  criteria  in  all  these  areas. 
Second,  the  award  fee  process  forces  up  front  planning  and 
this  planning  can  directly  and  Indirectly  affect  all  these 
areas.  Third,  the  type  of  contract  is  a  critical  factor  in 
the  effectiveness  of  an  award  fee  and  its  affect  on  these 
areas.  Lastly,  the  award  fee  criteria  may  incentivlze  a 
specific  area  at  the  expense  of  system  performance  and/or 
capability. 

Question  24  asked:  What  other  tools  does  the  program 
manager  have  besides  award  fee?  Each  interviewee  cited 
tools  that  are  described  in  the  FAR  and  its  supplements 
(withholding  of  payments,  or  cost  report  data),  standard 
reviews  and  audits  (design  reviews,  test  results,  or 
independent  verification  and  validation) ,  as  well  as 
specific  metrics  that  each  program  develops  during  the 
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length  of  the  program.  The  one  item  the  researcher  had  not 
found  in  any  literature  associated  with  software  development 
and  award  fee  was  the  contractor  performance  appraisal 
reporting  system  (CPARS).  Three  government  interviewees 
used  the  award  fee  performance  data  as  the  basis  for  their 
CPARS  reports.  This  allowed  the  government  progreun  manager 
additional  leverage  with  the  contractor  because  the  CPARS 
data  is  used  by  the  government  during  the  source  selection 
evaluation  process  for  future  contracts.  Government 
contractors  usually  are  very  interested  in  CPARS 
evaluations,  because  they  have  a  direct  effect  on 
contractors'  ability  to  win  future  systems  contracts.  Over 
fifty  (50)  percent  of  the  interviewees  that  responded  to  the 
question  referenced  the  systems  development  process,  with 
its  proper  implementation  and  continual  management  as  a  key 
tool  the  progreun  manager  has  to  manage  the  software 
development  process.  The  four  contractors  Interviewed  also 
cited  the  systems  development  process  and  its  proper 
management  as  a  key  tool  in  the  system  development  process. 

Question  25  asked:  How  has  having  the  award  fee  on 
your  program  benefitted  the  overall  cost,  schedule,  and 
performance?  The  interviewees  answers  Included:  it  has 
not,  $4  million  cost  overrun,  6  month  schedule  slip,  it  has 
helped  to  control  cost.  Improve  government  contractor 
relations  and  contractor  responsiveness  in  resolving 
problems  (Ai^endix  H,  pages  149-150).  The  specific 
interview  comments  to  question  twenty  five  are  in  Appendix  H 
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and  I.  To  present  the  data  from  this  question,  the 
researcher  divided  responses  Into  five  categories:  to  be 
determined  (TBD),  no  comment  (N/C),  focus  efforts,  no,  and 
help  process.  Figure  25  shows  the  distribution  of 
Interviewee  responses.  The  TBD  comment  came  from  an 
Interviewee  whose  contractor  had  just  started  work.  The  N/C 
was  from  a  Program  Executive  Officer  the  researcher 
Intexrvlewed  who  had  not  directly  managed  an  award  fee 
contract.  By  combining  the  responses  of  "focus  efforts"  and 
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Figure  25.  How  Has  Award  Fee  Benefitted  Your  Program? 


"help  process,"  the  data  supports  a  premise  that  award  fee 
contracts  have  a  positive  effect  on  systems  development, 
since  20  out  of  27  Interviewees  responded  positively. 
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In  question  26a  the  researcher  asked:  Do  you  plan  to 
have  an  award  fee  on  future  contracts?  Three  government 
interviewees  said  "no,"  and  one  government  interviewee  said 
"It  depends  and  if  applicable."  All  four  contractors 
interviewed  said  yes.  Figure  26  shows  the  responses  to  this 
question.  This  question  shows  23  out  of  27  interviewees 
would  use  award  fee  on  future  contracts. 
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Figure  26.  Award  Fee  on  Future  Contracts 


Question  26b  asked  "Why"  or  "why  not"  to  question  26a. 
The  question  26a  "No"  answers  seem  to  focus  on  the 
administrative  burden  of  award  fee  contracts.  These  are 
their  comments:  "too  much  administration  to  be  done,"  "the 
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process  used  to  get  to  the  FDO  [Fee  Determining  Official] 
has  too  many  layers/'  "the  fee  Is  determined  by  someone  too 
far  removed  from  the  process/'  "the  contracting  process  is 
flawed/'  and  "given  the  effectiveness  -  the  hassle  is  not 
worth  the  effort  -  too  much  government  overhead"  (Appendix 
H,  page  151).  The  "it  depends"  comment  in  26a  generated  a 
response  that  said  the  use  of  award  fee  depends  on  the 
system  and  its  specific  circumstances.  The  "yes"  government 
responses  to  26a  generated  key  words  for  affecting 
contractor  performance  like  motivate/  incentives/  effective 
tool/  and  leverage.  The  contractor  interviewees  cited 
motivation  and  the  belief  in  the  evaluation  process  as 
reasons  for  using  award  fee  in  the  future.  More  specific 
information  about  interviewee  reasons  for  using/  or  not 
using/  award  fee  can  be  found  in  Appendix  H  and  I. 

In  question  26c /  the  researcher  wanted  to  see  if 
software  development  would  be  included  in  future  award  fee 
contracts.  Two  government  interviewees  who  plan  to  use 
award  fee  in  the  future  said  it  would  not  include  software 
as  a  category.  Question  26d  was  used  to  allow  interviewees 
to  explain  their  decision.  All  contractor  interviewees  said 
"yes"  to  both  questions.  Figure  27  shows  the  interviewee 
responses . 

Question  26d  asked  "why"  or  "why  not"  would  software 
development  be  included  in  future  award  fee  contracts.  Two 
interviewees  that  said  "yes"  to  question  26a  and  "no"  to 
26c.  The  interviewees  gave  the  following  reasons  for  this 
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Figure  27.  Will  Future  Award  Fee  Contracts  Include 
Software  Development? 


change,  "Generally  don't  manage  our  contracts  at  that  level 
and  "No  further  software  development  Is  planned"  (Appendix 
H,  page  153).  These  answers  are  focused  on  the 
Interviewee's  specific  program  and  therefore  appear  not  to 
apply  to  all  software  development  programs  using  award  fee 
contracts.  The  interviewees  that  responded  "yes"  to 
questions  26a  and  26c  used  similar  reasons.  Two  reasons 
were  "The  award  will  make  the  software  development  managers 
responsive  to  procurement  agency  Inputs  and  concerns"  and 
"It  Is  a  better  way  to  get  a  contractor's  attention" 
(Appendix  H,  page  152-153).  Appendix  H  and  I  can  be 
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referred  to  gain  a  more  complete  understanding  of  the 
Interviewee  responses. 

Questions  27a  through  27c  focus  on  award  fee  being  a 
hindrance  to  government  and  contractors.  Question  27a 
asked:  Was  having  an  award  fee  on  you  program  a  hindrance 
to  overall  cost,  schedule,  and  performance?"  Twenty-six 
Interviewees  (96  percent)  said  "No"  (Figure  28).  Only  one 
Interviewee  said  "yes . " 
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Figure  28.  Was  Award  Fee  A  Hindrance  To  Your  Program? 


Question  27b  focused  on  why  the  Interviewees  responded 
to  question  27a  as  they  did.  The  Interviewees  that 
responded  "No"  (award  fee  was  not  a  hindrance)  cited  similar 
reasons  to  the  ones  they  gave  to  questions  25  and  26  In 
which  they  said  "Yes."  The  following  are  some  exeunples: 
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"Focused  contractor  on  areas  the  government  wanted,  in  this 
case  reliability,"  "Been  a  positive  influence,"  and  "I 
believe  their  initial  (first  period)  performance  would  have 
been  worse  without  award  fee"  (Appendix  H,  page  154).  The 
one  Intejr^rlewee  that  felt  award  fee  was  a  hindrance 
responded,  "Cost  the  government  time,  but  has  not  effected 
contractor  work"  (Appendix  H,  page  154). 

In  question  27c,  the  interviewees  were  asked,  "If  award 
fee  was  a  hindrance,  what  did  you  do  to  overcome  it?"  The 
only  Interviewee  to  comment  stated  the  following,  "It  was/is 
a  pain  to  administer  (for  government)  but  not  a  hindrance  to 
contractor"  (Appendix  H,  page  155).  The  interviewee  that 
said  award  fee  was  a  hindrance  (question  27a)  did  not 
comment  on  this  question.  Figure  29  shows  the  distribution 
of  responses. 

The  last  series  of  questions,  28a  through  28c,  were 
optional  and  focused  on  examples  of  award  fee  performance 
periods  and  evaluation  criteria.  Appendixes  A  and  B 
respectively.  Interviewees'  responses  to  these  questions 
fell  into  three  areas:  1)  They  answered  the  questions,  2) 
They  just  made  statements  about  Appendixes  A  and  B,  and  3) 
They  made  a  statement  about  award  fees  in  general. 

Question  28a  asked:  What  effect  on  the  process  of 
software  development  would  they  have?  Fourteen  (14)  of  27 
interviewees  responded  to  this  question.  Thirteen  (13)  of 
the  interviewees  thought  the  examples  in  Appendixes  A  and  B 
would  have  a  positive  effect  on  the  software  development 
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Figure  29.  If  Award  Fee  Was  A  Hindrance,  What  Did  You  Do  To 
Overcome  It? 


process.  Some  of  there  comments  were  "Positive,  better 
examples,  the  better  yours  will  be  done,  would  be  a  great 
help,"  "Yes  -  shows /communicates  what  Is  Important  and  how 
to  get  the  pot  of  gold  -  What  he  does  not  do  will  get  him 
zero,"  and  "Potential  of  Insuring  award  fees  are  done  right 
the  first  time"  (Appendix  H,  page  156).  One  Interviewee 
thought  Appendixes  A  and  B  would  have  limited  effect. 

In  question  28b  the  Interviewees  were  asked  "What 
benefit  (or  hindrance)  would  they  be  to  the  program  office 
and  the  contractor  If  made  part  of  the  request  for  proposal 
and  source  selection?"  Fourteen  (14)  of  the  27  Interviewees 
responded  to  this  question.  Thirteen  (13)  thought  Appendix 
A  and  B  would  be  beneficial  If  made  part  of  the  request  for 
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proposal  and  source  selection.  One  Interviewee  said  the 

earlier  the  better,  because  It  will  help  focus  contractor 

and  government  on  contractor  strengths  and  weaknesses 

(Appendix  H,  page  157).  Another  Interviewee  said: 

A  definite  benefit  to  Include  them  as  part  of  the  RFP 
package.  The  contractor  will  probably  be  receptive  to 
almost  any  type  of  award  fee  plan,  but  he  will  want  It 
to  be  something  that  Is  achievable.  An  award  fee  with 
criteria  that  are  Impossible  to  meet  does  nothing  for 
either  party.  (Appendix  H,  page  157) 

The  one  contractor  that  responded  had  thoughts  along  the 

same  line:  "It  would  be  beneficial  because  It  would  give 

the  proposal  manager/program  manager  a  heads  up  to 

understand  what  customer's  major  concerns  are"  (Appendix  I, 

page  171).  Government  Interviewee  1  thought  Appendix  B 

would  confuse  everyone.  This  Interviewee  did  not  further 

clarify  his  concerns  and  was  unavailable  to  the  researcher 

for  additional  clarification. 

Question  28c  asked  the  Interviewee  would  Appendix  A  and 

B  help  the  contractor  understand  what  must  be  accomplished 

to  receive  an  award  fee?  Fourteen  (14)  of  27  Interviewees 

responded  to  this  question,  with  Interviewee  1  being  the 

only  negative  response.  The  13  Interviewees  that  thought 

Appendixes  A  and  B  would  help  In  contractor  understanding 

cited  several  different  reasons.  One  Interviewee  thought 

the  contractor  would  focus  his  efforts  on  the  award  fee, 

which  may  be  good  or  bad  depending  on  the  program  Appendix 

H,  page  158).  This  focus  also  depends  on  how  the  award  fee, 

statement  of  work,  and  system  specification  are  relating  a 
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common  requirement  (Appendix  H,  page  158).  Another 
Interviewee  took  the  approach  that  "award  fee  will  force 
proper  up  front  planning  and  tracking  to  the  plan"  (Appendix 
H,  page  158).  Interviewee  1  believes  that  "communications 
between  the  government  and  contractor  is  key  to  a  successful 
program  and  award  fee  will  incentivize  this  process" 

Appendix  H,  page  158)  Interviewee  r  said  that  these 
Appendixes  could  possibly  help.  He  used  his  past  experience 
as  justification; 

Our  experience  with  our  contractor  revealed  that  when 
we  change  to  award  fee  criteria  similar  to  this  it 
caused  great  confusion  on  their  part.  They  saw  the 
criteria  as  a  checklist,  that  each  one  had  to  be  met 
before  moving  on  to  the  next  section.  Rather,  our 
Intent  was  to  take  observed  behavior  and  see  where  it 
fell  in  the  rating.  (Appendix  H,  page  158) 

The  one  contractor  to  respond  said  "they  would  summarize  the 
major  point  only"  and  the  contractor  program  managers  task 
is  to  "interpret  and  break  out  lower  level  concerns" 
(Appendix  I,  page  171).  As  in  question  28b,  interviewee  1 
said  Appendix  A  and  B  would  not  help  contractor  under¬ 
standing.  His  reason  was: 

An  award  fee  plan  is  by  its  nature  tailored  to  the 
specific  task/contract/program/phase.  These  may  be 
useful  as  lists  but  they  sure  should  not  be  dignified 
as  'policy'  or  regulation.  (Appendix  H,  page  158) 

The  researcher  asked  the  interviewees  if  they  had  any 

additional  comments  about  award  fee  or  the  Interview.  To 

this  question  the  researcher  received  16  responses.  The 

responses  received  fell  into  five  broad  categories: 

software,  criteria,  periods,  award  fee  plans,  and  training. 
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In  the  software  category,  government  interviewee  b 
suggested  a  definition  of  software  success  was  lacking.  He 
believes  tracking  completed  lines  of  code  (LOC)  can  be 
misleading  and  completed  modules  are  a  better  means  of 
tracking  software  success. 

Under  the  criteria  category,  six  government 
interviewees  had  comments.  Two  said  the  criteria  (Appendix 
B)  was  too  broad  and  contained  too  many  categories.  The  use 
of  so  many  criteria  makes  the  evaluation  too  fragmented. 
Government  interviewee  d  believes  the  criteria  needs  to 
focus  on  the  product  and  process  (Appendix  H,  page  159). 
Interviewee  m  questioned  the  criteria  (Appendix  B) 
definition  of  successful  completion.  He  said  no  design 
review  is  ever  100  percent  complete.  He  recommends  the 
award  fee  be  used  as  an  incentive  for  the  contractor  to 
exceed  contract  requirements  (Appendix  H,  page  159).  There 
were  two  contractor  interviewees  that  commented  about  the 
criteria.  These  contractors  were  concerned  that  the 
criteria  were  focused  only  on  software,  not  on  the  program 
as  a  whole.  They  (the  criteria)  need  to  be  very  program 
specific  to  attain  the  desired  end  item  and  not  that  he  (the 
contractor)  just  checked  the  square  (Appendix  I,  page  171). 

The  award  fee  periods  in  Appendix  A  brought  comments 
from  two  government  and  two  contractor  interviewees. 
Government  interviewees  d  and  f  had  opposing  views  on  what 
to  tie  award  fee  periods  to.  Interviewee  d  recommends 
award  fee  periods  be  tied  to  the  calendar,  not  milestones. 
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while  interviewee  £  recommends  periods  be  tied  to  milestones 
(Appendix  H,  page  159).  The  contractor  interviewees  focused 
on  number  and  length  of  the  award  fee  period.  Interviewee  b 
believes  four  to  five  award  fee  periods  are  plenty  to  attain 
the  desired  end  Item.  Interviewee  c  believes  periods  should 
be  six  months  In  length  and  be  a  "snap  shot  of  the  next  two 
or  more  events"  (Appendix  1,  page  171-172). 

The  award  fee  category  comments  come  from  six 
government  and  two  contractor  Interviewees.  Interviewees  k 
and  p  focused  on  the  EDO's  role  In  the  award  fee  process. 
Interviewee  k  believes  the  EDO  should  be  the  two  letter 
Program  Director.  His  reason  was  the  Progreun  Director  has 
responsibility  for  program  and  should  have  EDO  authority 
(Appendix  H,  page  159).  Interviewee  p  felt  the  EDO  changed 
too  many  times  and  there  was  no  continuity  In  the  evaluation 
process  (Appendix  H,  page  160).  Interviewee  f  believes  no 
matter  how  good  the  award  fee  plan  Is,  It  cannot  fix  a  bad 
statement  of  work  or  contract  (Appendix  H,  page  159). 
Inteirvlewee  o  feels  the  "working  relationship  between 
government  and  contractor  Is  more  important  than  award  fee" 
(Appendix  H,  page  160).  A  very  different  approach  was 
presented  by  Interviewee  s.  He  recommends  the  award  fee 
pool  be  used  to  "help  contractor  with  capitalization  by 
giving  him  award  fee  up  front"  (Appendix  H,  page  160).  If 
the  contractor  does  not  earn  100  percent  of  the  period's  fee 
the  contractor  must  pay  the  government  back.  Interviewee  s 
states  the  hlnderance  as  "this  approach  'drives  PCO  crazy'" 
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(Appendix  H,  page  160).  Interviewee  t  believes  "whatever 
guidance  Is  provided.  It  must  be  able  to  be  tailored  to  meet 
the  needs  of  the  program  It  Is  being  applied  to"  (Appendix 
H,  page  160).  The  two  contractor  interviewees  b  and  d 
focused  on  different  ares  from  each  other  and  the  government 
interviewees.  Interviewee  b  believes  that  "award  fee 
percentage  versus  ratings  Is  Important  and  should  be  shown," 
while  Interviewee  d  feels  "praise  or  corrective  action 
Instituted  by  the  award  percentage  . . .  determines  how  much 
management  'help'  Is  supplied."  (Appendix  I,  page  171-172). 

The  last  category  was  training.  Government 
Interviewees  g  and  p  believe  there  Is  a  lack  of  award  fee 
trained  personnel  in  the  field  today.  Interviewee  p 
believes  a  lessons  learned  data  base  Is  needed  and  may  be  a 
start  In  filling  this  void  (Appendix  H,  page  160). 

Summary 

This  chapter  presented  a  summary  of  the  Interview  data 
contained  In  Appendix  H  and  I.  Not  all  Interviewee  comments 
were  covered  In  this  siimmary,  however.  In  the  opinion  of  the 
researcher,  not  all  were  focused  on  the  research  questions 
presented  In  Chapter  I.  The  data  presented  In  this  chapter 
should  give  the  reader  an  Idea  of  Program  Director  thoughts 
on  award  fee  contract. 
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Chapter  V  will  review  the  research  questions  presented 
in  Chapter  I  using  both  literature  and  interview  data,  and 
provide  the  researcher's  recommended  answers.  The  chapter 
will  close  with  a  recommendation  for  future  research. 
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V.  Recommendations 


Introduction 

This  chapter  gathers  the  information  in  the  Literature 
Review  (Chapter  2)  and  Findings  (Chapter  4),  and  presents 
the  researcher's  beliefs  concerning  the  research  problem 
statement  and  research  question  in  Chapter  1.  The  data 
obtained  in  Chapter  4  was  obtained  from  interviewing  23 
government  and  4  contractor  program  directors  and  deputies. 
The  government  Interviews  were  conducted  with  personnel  from 
Aeronautical  Systems  Center  (old  Aeronautical  Systems 
Division),  Electronic  Systems  Center  (old  Electronic  Systems 
Division),  and  Space  and  Missile  Systems  Center  (old  Space 
Systems  Division) .  The  contractor  interviews  were  conducted 
with  personnel  the  researcher  was  able  to  contact  through 
the  program  offices  interviewed. 

Research  Findings 

The  focus  of  this  research  was  on  finding  the 
contractual  tools  available  to  influence  contractor 
performance  during  the  software  development  process.  Five 
research  questions  were  developed  to  guide  the  literature 
and  questionnaire  data  collection  process.  The  following 
section  states  the  research  questions  and  discusses  the 
research  Findings. 

Question  It  What  provisions  are  available  in  the 
Federal  Acquisition  Regulation  (FAR)  and  its  supplemental 
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regulations  to  influence  contractor  performance  during  the 
development  process? 

Findings ;  The  literature  search  found  the  type  of 
contract  used  as  a  key  FAR  management  tool  available  to 
influence  contractor  performance  during  the  software 
development  process.  Award  fee  was  found  to  be  the  most 
flexible  provision  in  the  FAR  and  its  supplements  to 
influence  contractor  performance  during  the  development 
process.  Award  fee  also  provides  an  appropriate  vehicle  for 
feedback. 

Question  2;  What  events  or  milestones  in  the 
development  process  should  be  used  to  evaluate  contractor 
performance  during  the  software  development  process? 

Findings ;  Articles  and  books  presented  several 
approaches  for  events  or  milestones  in  the  software 
development  process  that  could  be  used  to  evaluate 
contractor  performance.  The  literature  search  found  a 
strong  consensus  on  major  events  or  milestones  in  the 
software  development  process. 

There  were  nine  common  events  or  milestones  within  each 
of  the  readings  that  can  be  used  to  evaluate  contractor 
perfozmiance.  The  consensus  of  milestones  that  should  be 
used  are  systems  requirements  review  (SRR),  system  design 
review  (SDR),  software  specification  review  (SSR), 
prellmlnazry  design  review  (FDR),  critical  design  review 
(CDR),  test  readiness  review  (TRR),  functional  configuration 
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audit  (FCA),  physical  configuration  audit  (PCA),  and  formal 
qualification  review  (FQR).  These  specific  names  were  not 
used  by  all  the  authors  to  describe  the  events  in  their 
software  development  life  cycle,  nor  do  all  the  phases  fit 
every  Government  software  development  project.  Yourdon 
states  "certainly  every  project,  whether  structured  or  not, 
goes  through  some  kind  of  system  analysis,  design,  and 
implementation"  (Yourdon,  1988:45).  These  nine  events  and 
milestones  should  be  tailored  for  each  software  development 
project. 

Question  3:  How  should  award  fee,  as  described  in  the 
FAR  and  its  supplements,  and  software  development  events  be 
linked  to  Influence  contractor  performance? 

Finding:  The  interviews  conducted  did  not  present  a 
clear  consensus  on  the  linkage  of  software  development 
events  and  award  fee.  The  periods  in  Appendix  A  brought 
comments  from  two  government  and  two  contractor  interviews. 
Government  interviews  d  and  f  had  opposing  views  on  what  to 
tie  award  fee  periods  to.  Interviewee  d  recommends  that 
award  fee  periods  be  tied  to  calendar  dates  not  milestones; 
Interviewee  f  recommends  periods  be  tied  to  milestones 
(Appendix  H,  page  159).  The  contractor  interviews  focused 
on  number  and  length  of  award  fee  period.  Interviewee  b 
believes  four  to  five  award  fee  periods  are  adequate  to 
attain  the  desired  end  Item  (Appendix  I,  page  171). 
Interviewee  c  believes  time  periods  should  be  six  months  In 
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length  and  be  a  "snap  shot  of  the  next  two  or  more  events" 
(Appendix  I,  page  171).  It  appears  to  the  researcher  that 
program  requirements  are  the  key  factor  that  drives  the 
linkage  between  award  fee  and  the  software  development 
events  outlined  In  Question  2  Findings  above.  The  reason 
the  researcher  makes  this  statement  Is  that  In  each 
Interview  referenced,  the  award  fee  process  Is,  or  has 
worked,  with  some  measure  of  success.  The  example  In 
Appendix  A,  Performance  Evaluation  Periods.  Is  a  good 
starting  point  for  discussion. 

Question  4;  Can  the  award  fee  process  be  used  to 
Influence  contractor  performance  during  software 
development? 

Findings;  The  literature  reviewed  did  not  give  a  clear 
perspective  on  this  question.  To  answer  this  question,  the 
researcher  used  Interview  questions  16  through  23.  These 
questions  focused  on  the  affect  award  fee  would  have  on 
software  development,  cost,  delivery,  reliability, 
documentation,  testability,  and  quality.  The  Interviews 
found  88  percent  of  the  respondents  believe  award  fee  has  a 
positive  affect  on  contractor  software  development  and  92 
percent  of  the  respondents  felt  award  fee  could  affect 
contractor  performance  on  software  documentation;  however, 
the  Interviews  also  found  no  clear  consensus  on  the  affect 
award  fee  would  have  on  cost,  delivery,  reliability, 
testability,  and  quality. 
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Question  5;  What  award  fee  criteria  should  be  used  to 


evaluate  contractor  performance? 

Findings ;  The  exact  criteria  is  dependent  on  the 
specific  software  development  program.  The  researcher  used 
the  perspectives  of  current  and  former  program  directors  to 
find  the  current  system  program  office  philosophy  on 
criteria.  The  researcher  used  interview  guestlon  28  and 
Appendix  B  to  gain  some  insight  into  current  criteria  used 
in  system  program  offices.  Ninety-two  percent  of  the 
responding  interviewees  felt  the  criteria  presented  in 
Appendix  B  would  be  "Positive,  better  examples,  the  better 
yours  will  be  done,  would  be  a  great  help"  (Appendix  H,  page 
156).  Two  government  interviewees  said  the  criteria 
(Appendix  B)  was  too  broad  and  contained  too  many 
categories.  They  believe  the  use  of  so  many  criteria  makes 
the  evaluation  too  fragmented.  The  two  contractors  that 
responded  were  concerned  the  criteria  were  focused  only  on 
software,  not  on  the  program  as  a  whole  and  believe  the 
criteria  should  be  very  program  specific  to  attain  desired 
end  item  [not  just  checking  squares]  (Appendix  I,  page  171). 

The  researcher  believes  the  criteria  in  Appendix  B  can 
be  used  as  a  starting  point  to  focus  the  development  of 
future  award  fee  contract  criteria.  It  is  not  intended  to 
be  blindly  applied  to  all  award  fee  contracts.  The  criteria 
in  Appendix  B  is  only  intended  to  be  the  starting  point  and 
should  be  specifically  tailored  to  the  needs  of  each 
program. 
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Question  6 :  Does  the  award  fee  process  Include  an 


appropriate  type  and  form  of  feedback  to  give  to  the 
contractor  for  Improving  his  ability  to  develop  maintainable 
software  on  time,  and  within  cost? 

Findings :  The  literature  did  not  fully  answer  this 
question.  The  researcher  made  an  attempt  to  focus  Interview 
questions  28b  and  28c  to  answer  this  question.  The  13 
respondents  felt  that  award  fee  focuses  the  effort  of  both 
the  government  and  contractor.  They  believe  the  award  fee 
process  fosters  communications  between  the  two  and  this 
communication  "Is  key  to  a  successful  program  and  award  fee 
will  Incentlvlze  this  process”  (Appendix  H,  page  158). 

Based  on  the  data  from  the  Interviews,  the  researcher 
believes  the  award  fee  process  Includes  the  appropriate  type 
and  form  of  feedback  for  a  contractor  to  Improve  his  ability 
to  develop  maintainable  software  on  time,  and  within  cost. 
The  key  Is  communication  and  award  fee  fosters  It. 

Research  Summary 

This  research  has  focused  on  the  contractual  tools 
available  to  Influence  contractor  performance  during  the 
development  process.  The  research  found  award  fee  Is  the 
tool  that  can  be  tailored  to  a  programs  specific  need  and 
allows  the  progreun  director  to  refocus  the  contractor 
efforts.  Award  fee  Is  like  any  tool.  It  must  be  developed 
and  executed  correctly  to  have  the  desired  effect.  The 
award  fee  Is  only  one  part  of  the  acquisition  process  and  no 
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matter  how  good  the  award  fee  plan  is,  it  cannot  fix  a  bad 
statement  of  work  or  contract  (Appendix  H,  page  159.) . 

During  the  course  of  this  research,  the  award  fee  plan 
for  the  F-22  System  Program  Office  was  found  to  be  an 
excellent  example.  It  is  Included  in  this  thesis  as 
Appendix  J  for  future  reference  by  the  reader. 

Recommendation  for  Future  Research 

As  a  result  of  this  research,  three  areas,  in  the 
researcher's  opinion,  require  further  investigation.  First, 
the  area  of  experience  of  the  acquisition  force  with  award 
fees  should  be  investigated.  The  researcher  found  that  all 
program  directors  interviewed  had  at  least  some  experience 
with  award  fee  contracts.  Their  concern  was  that  the 
personnel  within  their  program  did  not  have  much,  if  any, 
award  fee  experience  before  the  award  fee  process  was 
started.  By  finding  the  program  office  experience  base  with 
award  fee,  an  award  fee  training  program  may  be  justified 
and  initiated.  Second,  the  requirements  for  a  standardized 
award  fee  training  progreun  should  be  investigated.  This 
training  program  would  increase  the  acquisition  corps 
knowledge  base  on  award  fee  and  enable  the  development  of  an 
award  fee  data  base  that  would  assist  in  the  development  of 
future  award  fee  contracts.  The  last  area  is  the  use  of 
award  fee  as  a  tool  to  achieve  an  end.  The  concept  of  ends 
and  means  has  been  researched  before  (Quinn  and  Rohrbaugh, 
1983:363-377).  The  purpose  of  this  investigation  would  be 
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to  examine  the  relationship  of  end  and  means  in  the  context 
of  award  fees.  The  question  is:  are  we,  the  government, 
giving  the  contractor  the  appropriate  means  to  achieve  the 
desired  ends  in  the  award  fee  process? 
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(ESC,  1990b:  Annex  4) 


The  evaluation  periods  for  this  contract  shall  be  tied  to 
the  program's  contractual  baseline.  The  contractual  baseline 
shall  be  identified  by  contract  award  and  shall  be  updated 
as  required  with  government  review  and  approval. 

Performance  periods  1  through  5  will  use  Performance 
Evaluation  Areas  A,  B,  and  D.  Performance  periods  6  through 
9  will  use  Performance  Evaluation  Areas  A,  B,  C,  and  D. 

FEE  ALLOCATION  (Sample  Plan) 

PERIOD  %  START  OF  PERIOD  END  OF  PERIOD 

1.  5  Start  of  Contract  Successful  completion  of 

System  Requirements  Review 
(SRR)  to  include 
disposition  of  Contractor's 
open  action  items  and 
delivery  of  final  SRR 
documentation  to  the  Air 
Force . 

2.  15  End  of  Period  1  Successful  completion  of 

Software  Specification 
Review  (SSR)  to  include 
disposition  of  contractor's 
open  action  items  and 
delivery  of  final  SSR 
documentation  to  the  Air 
Force . 

3.  10  End  of  Period  2  Successful  completion  of 

the  last  Preliminary  Design 
Review  (PDR)  to  include 
disposition  of  contractor's 
open  action  items  and 
delivery  of  last  PDR 
documentation  to  the  Air 
Force . 

4.  20  End  of  Period  3  Successful  completion  of 

the  last  Critical  Design 
Review  (CDR)  to  Include 
disposition  of  contractor's 
open  action  items  and 
delivery  of  last  CDR 
documentation  to  the  Air 
Force . 
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PERIOD  %  START  OF  PERIOD 

5.  10  End  of  Period  4 


6.  5  End  of  Period  5 


7.  15  End  of  Period  6 


8.  10  End  of  Period  7 


9.  10  End  of  Period  8 


Total 

%100 


END  OF  PERIOD 

Successful  completion  of 
Progreun  Status  Review  (PSR) 
and  delivery  of  all  PSR 
documentation  to  the  Air 
Force . 

Successful  completion  of 
Progreun  Status  Review  (PSR) 
and  delivery  of  all  PSR 
documentation  to  the  Air 
Force . 

Successful  completion  of 
Test  Readiness  Review  (TRR) 
to  Include  disposition  of 
contractor's  open  action 
Items  and  delivery  of  final 
TRR  documentation  to  the 
Air  Force. 

Successful  completion  of 
Functional  Configuration 
Audit  (FCA)  and  Physical 
Configuration  Audit  (PCA) 
to  Include  disposition  of 
contractor'  open  action 
Items  and  delivery  of  final 
Functional  Configuration 
Audit  and  Physical 
Configuration  Audit 
doctimentatlon  to  the  Air 
Force . 

Successful  completion  of 
final  Operational  Test  and 
Evaluation  (OT&E)  to 
Include  disposition  of 
contractor's  open  action 
Items  and  delivery  of  final 
OT&E  documentation  to  the 
Air  Force. 
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Appendix  B;  Evaluation  Criteria 
(ESC,  1990b:  Annex  3) 


NOTE:  THE  FOLLOWING  ARE  ONLY  EXAMPLES  AND  ARE  NOT  MEANT 
TO  BE  ALL  INCLUSIVE. 


Area  A;  General  Management  Practices 


Unacceptable  Rating  (0-59  points K  The  contractor 
shall  earn  this  rating  and  associated  points  for  the 
following: 

1.  Causes  of  significant  problems  were  identified  late. 
Solutions  were  only  stop-gap  measures  and  little  attempt  was 
made  to  define  more  effective  options  and  solutions. 

2.  In-depth  risk  analysis  was  lacking. 

3.  Poor  corrective  action  was  taken  on  identified 
problems . 

4.  Evaluation  of  problems  was  not  presented  to  the  program 
office  in  a  timely  fashion. 

5.  Program  visibility  was  lacking. 

6.  Proposals  were  consistently  late  and  incomplete, 
impacting  audlt/f act-finding  efforts. 

7 .  Poor  communication  of  program  issues  in  meetings  and 
presentations . 

8.  Contract  Deliverables  (CDRLs)  were  frequently  late, 
inadequate,  and  requiring  corrections. 

9.  Poor  visibility  of  progreun  and  equipment  costs. 

10.  Program  schedules  were  out-of-date,  do  not  show 
critical  path  and  do  not  show  changes  from  Government 
approved  baseline. 

11.  Lack  of  manning  with  appropriately  trained/cleared 
personnel . 

12.  Disregard  for  cost  effectiveness  in  management 
decisions. 

13.  C/SSR  deviation  were  unacceptable. 
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14.  Schedule  milestones  slippage  was  significant  and 
without  warning. 

15.  The  scope  and  extent  of  documentation  changes  was 
unacceptable . 

16.  The  Contractor  was  not  timely  and  responsive  in  acting 
upon  Government  program  office  requirements. 

Good  Rating  (60  -  79  points K  The  contractor  shall  earn 
this  rating  and  associated  points  for  the  following: 

1.  Timely  identification  of  problem  causes  and  solutions. 

2.  Applies  risk  analysis  to  assess  total  impact  and 
minimize  subsequent  impact. 

3.  Timely  and  aggressive  corrective  action  on  identified 
problems  was  lacking. 

4.  Meets  critical  milestones. 

5.  Provides  timely  evaluation  of  problems  to  progreun 
office. 

6.  Maintains  visibility  of  progreun  and  equipment  costs  and 
schedules . 

7.  Proposals  were  submitted  on  time  with  consistent 
quality  and  no  major  audit/ fact> finding  and  negotiation 
schedule  problems. 

8.  Program  Issues  were  communicated  in  a  timely  manner, 
minimizing  program  Impacts. 

9.  CDRLs  were  submitted  on  time  with  consistent  quality. 

10.  Maintains  visibility  of  program  and  equipment  costs. 

11.  Program  schedules  were  up-to-date,  show  critical  path 
and  show  changes  from  Government  approved  baseline. 

12.  Consistent  level  of  manning  with  appropriately  trained 
personnel . 

13.  Cost  effectiveness  was  part  of  management  decisions. 

14.  C/SSR  deviation  were  frequent. 

15.  Documentation  changes  were  extensive  but  manageable. 

16.  The  Contractor  was  somewhat  timely  and  responsive  in 
acting  upon  Government  progreun  office  requirements. 
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very  Good  Rating  f80  -  89  points K  The  contractor  shall 
earn  this  rating  and  associated  points  if  the  majority  of 
the  'Good'  criteria  were  satisfied,  plus  the  following: 

1.  Anticipates  problems  and  takes  corrective  actions  to 
minimize  impact. 

2.  Solutions  demonstrate  initiative  and  require  little 
revision. 

3.  Required  changes  implemented  with  minor  Impact. 

4.  Request  data  for  early  problem  anticipation. 

5.  Meets  all  significant  schedule  milestones. 

6.  Applies  clear  cost/effective  trade-offs  to  meet 
performance  requirements  at  minimum  cost. 

7.  Proposals  were  submitted  on  time  with  superior 
documentation  and  no  audit  concerns. 

8.  CDRLs  were  submitted  on  time  and  demonstrated  superior 
subject  knowledge. 

9.  Skillfully  adjusts  program  and  equipments  schedule 
priorities  based  on  critical  path  analyses. 

10.  Supports  subcontractor (s)  in  a  timely  and  effective 
manner  and  contributes  meaningful  Inputs. 

11.  Responsive  and  efficient  communication  of  program 
issues  with  program  office. 

12.  Appropriately  trained  personnel  manning  was  not  a 
problem. 

13.  Cost  effectiveness  was  key  part  of  management 
decisions . 

14.  C/SSR  deviation  were  infrequent. 

15.  Schedule  milestones  showed  only  minor  slippage  from  the 
Government  approved  baseline.  Re-plans  and  presents 
recommendations  to  the  Government  to  bring  the  schedule  in 
line. 


16.  Documentation  changes  were  normal  and  well  managed. 

17.  The  development  never  has  problems  with  an  adequate 
supply  of  trained  personnel. 
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18.  The  Contractor  was  timely  and  responsive  In  acting  upon 
Government  program  office  requirements. 

Excellent  Rating  f90  -  100  points K  The  contractor  shall 
earn  this  rating  and  associated  points.  If  the  majority  of 
the  'Very  Good'  criteria  were  satisfied,  plus  the  following; 

1.  Solves  difficult  Interface  problems  and  Implements  low 
cost,  high  performance  solutions  which  significantly  enhance 
overall  progreun  success. 

2.  Enthusiastically  Implements  cost  savings  Ideas  which 
result  In  system  cost  reductions. 

3.  Takes  the  Initiative  to  work  out  difficult  problems 
with  subcontractor(s) . 

4 .  Demonstrates  clear  corporate  commitment  to  program 
goals  and  objectives. 

5.  Communications  with  program  office  demonstrate 
Initiative  and  teamwork  In  solving  program  Issues. 

6 .  Proposals  submitted  early  and  negotiations  completed 
ahead  of  schedule. 

7.  C/SSR  deviation  were  almost  non-existent. 

8.  Schedule  milestones  were  maintained  at  the  Government 
approved  baseline. 

9.  Documentation  changes  were  minimized  to  the  lowest. 

10.  The  Contractor  was  very  timely  and  responsive  In  acting 
upon  Government  program  office  requirements. 
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Area  B:  Systems  Engineering 


Unacceptable  Rating  fO  -  59 K  The  contractor  shall  earn 
this  rating  and  associated  points  for  the  following: 

1.  Documentation  delivered  for  Government  formal  review 
was  Incomplete,  Inaccurate  and  contain  untestable 
requirements . 

2.  Formal  reviews  were  Incomplete,  and  Inaccurate. 

3.  Documentation,  hardware  and  software  don't  demonstrate 
requirements  traceability. 

4.  Contractor  does  not  use  modular  and  reusable  design  and 
programming  practices . 

5.  Contractor  does  not  follow  or  update  the  System 
Engineering  Management  Plan  and  Software  Development  Plan. 

6.  Develops  and  delivers  software  code  other  than  ADA 
without  formal  Government  waiver. 

7.  No  Government  approved  software  development  tools  were 
used. 


8.  Falls  to  provide  accurate  and  complete  list  of  proposed 
spares . 

9.  Growth,  from  approved  Government  baseline.  In  code, 
response  time,  and  throughput  capacity  (memory,  disk,  CPU 
utility)  were  consistently  greater  than  10%  or  greater  than 
15%  In  a  period. 

10.  Human  Factors  Engineering  (HFE)  and  Reliability, 
Maintainability,  and  Availability  (RMA)  were  not  considered. 

11.  The  Data  Accessions  List  was  meaningless  and  not 
updated . 

12.  Software  units  perform  multiple  tasks/functions  with  no 
clear  Inputs  and  outputs. 

13.  Software  metrics  were  not  maintained  or  accurate. 

14.  Quality  Assurance  (QA)  was  not  used  or  demonstrated. 

15.  System  Design  was  Inadequate  or  does  not  meet 
requirements . 


93 


Good  Rating  (60  -  79  polntsK  The  contractor  shall  earn 
this  rating  and  associated  points  for  the  following: 

1.  Documentation  delivered  for  Government  formal  review 
was  complete  and  accurate. 

2.  Formal  reviews  were  complete  and  accurate. 

3.  Documentation,  hardware,  and  software  were  developed 
and  delivered  to  the  Government  with  traceable  requirements. 

4.  Contractor  uses  modular  and  reusable  design  and 
programming  practices. 

5.  Contractor  follows  or  updates  the  System  Engineering 
Plan  and  Software  Development  Plan. 

6.  Contractor  develops  and  delivers  software  code  In  ADA, 
unless  waiver  was  formally  approved  by  the  Government. 

7.  Government  approved  software  tools  were  used. 

8.  Final  documentation  reflects  delivered  hardware  and 
software . 

9.  Follows  configuration  management  procedures  outlined  in 
Software  Development  Plan. 

10.  Operational  Implementation  of  designs  were  presented  to 
the  Government. 

11.  Contractor  adequately  prepares  for  Design  Reviews, 
Technical  Interchange  Meetings,  and  Tests/Installatlons. 

12.  Provides  a  complete  list  of  proposed  spares. 

13.  Growth,  from  approved  Government  baseline.  In  code, 
response  time,  and  throughput  capacity  (memory,  disk,  CPU 
utility)  were  consistently  greater  than  10%  but  less  than 
15%  In  a  period. 

14.  HFE  and  RMA  were  considered. 

15.  The  Data  Accessions  List  was  meaningful  and  up  to  date. 

16.  Software  units  perform  single  task/function  with  clear 
Inputs  and  outputs. 

17.  Software  metrics  were  maintained. 

18.  QA  was  used. 
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19.  System  Design  was  barely  adequate  and  meets  most 
requirements . 

Very  Good  Rating  (80  -  89  points K  The  contractor  shall 
earn  this  rating  and  associated  points  If  the  majority  of 
the  "Good"  criteria  were  satisfied  plus  the  following: 

1.  Documentation  delivered  for  Government  review  contains 
no  untestable  requirements. 

2.  Formal  reviews  were  comprehensive. 

3.  Requirements  were  easily  traced  In  all  dociimentatlon 
and  Design  Reviews. 

4.  Modular  and  reusable  design  and  programming  practices 
were  used  throughout  this  development  unless  a  formal  waiver 
was  granted  by  the  Government. 

5.  The  System  Engineering  Plan  and  Software  Development 
Plan  reflect  the  day  to  day  operations  throughout  this 
development. 

6.  ADA  software  code  was  developed  with  a  software  library 
In  mind. 

7.  Government  approved  software  tools  were  used  to  the 
maximum  extent  possible 

8.  Documentation  reflects  delivered  hardware  and  software 
at  delivery. 

9.  Operational  Implementation  was  considered  throughout 
the  development  and  delivery  process. 

10.  Contractor  effectively  prepares  for  Design  Reviews, 
Technical  Interchange  Meets  (TIM's)  and  Tests/Installatlons . 

11.  Provides  a  list  of  proposed  spares  ahead  of  schedule. 

12.  List  of  proposed  spares  was  complete  and  requires  no 
more  than  minor  alteration  by  Government  auditors. 

13.  Growth,  from  approved  Government  baseline.  In  code, 
response  time,  and  throughput  capacity  (memory,  disk,  CPU 
utility)  were  consistently  greater  than  5%  but  less  than  10% 
In  a  period. 

14.  HFE  and  RMA  were  shown  to  be  a  driver. 

15.  Software  units  perform  single  well  defined 
task/ function  with  clear  Inputs  and  outputs. 
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16.  Software  metrics  were  maintained  and  accurate. 

17.  QA  was  used  and  demonstrated  to  the  Government. 

18.  System  Design  was  adequate  and  meets  requirements. 

Excellent  Rating  (90  -  lOQ  points).  The  contractor  shall 
earn  this  rating  and  associated  points  if  the  majority  of 
the  'Very  Good'  criteria  were  satisfied  plus  the  following: 

1.  No  untraceable  requirements  were  found. 

2.  Contractor  uses  effective/efficient  modular  and 
reusable  design/programming  practices. 

3.  Good  System  Engineering  practices  were  used  throughout 
the  development  process. 

4.  Good  Configuration  Management  practices  were  used 
throughout  the  development  process. 

5.  Government  approved  automated  development  tools  were 
used  to  the  maximum  extent  possible. 

6.  Provides  a  thorough  list  of  proposed  spares  based  on 
projected  MTBF. 

7.  List  of  proposed  spares  was  provided  requiring  no 
alteration  by  Government  auditors. 

8.  Growth,  from  approved  Government  baseline,  in  code, 
response  time,  and  throughput  capacity  (memory,  disk,  CPU 
utility)  were  consistently  less  than  or  equal  to  5%  in  a 
period. 

9.  HFE  and  RMA  were  shown  to  be  a  driver  and  considered 
appropriately . 

10.  System  Design  optimizes  resources  and  exceeds 
requirements . 
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Area  C;  Test.  Evaluation  and  Installation 


Unacceptable  Rating  fO  -  59  points K  The  contractor 
shall  earn  this  rating  and  associated  points  for  the 
following: 

1.  Contractor  Configuration  Management  impacts  Test, 
Evaluation  and  Installation. 

2.  System  not  ready  for  formal  Acceptance  Testing  (e.g.. 
Boxes  not  Buttoned  up.  Functional  Configuration  Audit 
failure,  etc). 

3.  Inexperienced  Contractor  Test,  Evaluation,  and 
Installation  personnel  used. 

4.  Physical  Configuration  Audit  failure. 

5.  Government  approved  Test,  Evaluation,  and  Installation 
procedures  not  used. 

6.  Execution  of  unrealistic  Test,  Evaluation,  and 
Installation  schedules  used. 

7.  Use  of  unrealistic  plans  and  procedures. 

8.  Failure  to  supply  appropriate  packing  lists. 

9.  No,  or  ineffective,  operator  and  maintenance  training 
was  performed. 

Good  Rating  ^60  -  79  points).  The  contractor  shall  earn 
this  rating  and  associated  points  for  the  following: 

1.  Contractor  Configuration  Management  does  not  impact 
Test,  Evaluation,  and  Installation. 

2 .  System  was  ready  for  formal  Acceptance  Testing  ; 3 . g . , 
Boxes  were  Buttoned  up.  Functional  Configuration  Audit 
passed,  etc) . 

3.  Well  qualified  Contractor  Test,  Evaluation,  and 
Installation  personnel  were  used. 

4.  Physical  Configuration  Audit  passed  with  no  major 
problems . 

5.  Government  approved  Test,  Evaluation,  and  Installation 
procedures  were  used. 

6.  Execution  of  Test,  Evaluation,  and  Installation 
schedules . 
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7 .  Use  of  Government  approved  plans  and  procedures . 

8.  Contractor  supplied  packing  lists. 

9.  Operator  and  maintenance  training  was  performed. 

Very  Good  Rating  f80  -  89  points).  The  contractor  shall 
earn  this  rating  and  associated  points  if  the  majority  of 
the  "Good"  criteria  were  satisfied  plus  the  following: 

1.  Contractor  Configuration  Management  was  key  to  the 
success  of  Test,  Evaluation,  and  Installation. 

2.  System  was  well  prepared  for  formal  Acceptance  Testing. 

3.  Experienced  Contractor  Test,  Evaluation,  and 
Installation  personnel  used. 

4.  Physical  Configuration  Audit  passed  with  only  minor 
problems . 

5.  Government  approved  Test,  Evaluation,  and  Installation 
procedures  were  correctly  and  effectively  used. 

6.  Execution  of  realistic  Test,  Evaluation,  and 
Installation  schedules. 

7.  Use  of  realistic  plans  and  procedures. 

8.  Contractor  supplied  appropriate  packing  lists. 

9.  Effective  operator  and  maintenance  training  was 
performed. 

Excellent  Rating  (90  -  100  points).  The  contractor  shall 
earn  this  rating  and  associated  points  if  the  majority  of 
the  'Very  Good'  criteria  were  satisfied  plus  the  following: 

1.  Functional  Configuration  Audit  passed  with  no  problems. 

2.  Physical  Configuration  Audit  passed  with  no  problems. 

3.  No  problems  were  found  during  installation. 

4.  Excellent  operator  and  maintenance  training  was 
performed. 
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Area  D:  Security 


Unacceptable  Rating  (0-  59  points).  The  contractor  shall 
earn  this  rating  and  associated  points  for  the  following: 

1.  Contractor  Security  Program  does  not  comply  with  the 
Industrial  Security  Program  and/or  the  contract  DD  254 
and/or  the  applicable  classification  guides. 

2.  Contractor  does  not  have  a  formal  continuing  Security 
Training  program  for  contractor/subcontractor  personnel. 

3.  Inexperienced  Contractor  Security  personnel  used. 

4.  Failure  of  an  Security  inspection  and/or  inspection. 

5.  Documentation  was  not  controlled  in  accordance  with 
Industrial  Security  requirements. 

6.  No  clear  lines  of  authority  and  responsibility  were 
established  within  the  security  organization. 

7.  Staffing  was  not  adequate  to  accomplish  the  tasks 
required  by  security  directives  and  the  needs  of  the  program 
in  a  timely  manner. 

8.  Communications  were  not  adequate  to  facilitate  timely, 
responsive  replies  to  oral  and  written  communications. 

9.  Security  management  staff  does  not  plan  ahead  for  all 
security  and  related  program  elements  to  provide  staff, 
material  and  facility  resources  to  accomplish  projected 
requirements . 

10.  Billet  structure  will  not  meet  current  and  projected 
future  program  requirements. 

11.  The  Contractor  does  not  furnish  brief /debrief 
statements  in  a  timely  manner  and  does  not  avoid  double 
billeting  by  assuring  that  existing  personnel  were  debriefed 
in  a  timely  manner. 

12.  The  Contractor  does  not  process  access  requests  to 
appropriate  Government  agencies  in  a  timely  manor:  does  not 
track  progress  and  report  status  to  cognizant  Government 
program  and  security  management. 

13.  The  Contractor  was  not  timely  in  development  and 
revising  program  unique  Security  Practices  and  Procedures 
(SPP's)  and  adherence  to  approved  SPP's. 
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14 .  Contractor  does  not  provide  necessary  support  to 
acquire  accreditation  or  approval  of  facilities  for  special 
program  requirements. 

15.  Contractor  does  not  adhere  to  the  security  guides  and 
systems  of  control  number  assignment  to  produce  a  system  of 
accountability  that  provides  absolute  traceability  and 
minimizes  possible  compromise  of  program  material. 

16.  The  Contractor  does  not  take  all  necessaxry  steps  to 
eliminate  security  Infractions,  conduct  and  document 
Investigations  when  necessary,  and  does  not  prevent 
recurrences . 

Good  Rating  f60  -  79  points K  The  contractor  shall  earn 
this  rating  and  associated  points  for  the  following: 

1.  Contractor  Security  Program  has  many  deviations  from 
the  Industrial  Security  Program  and/or  the  contract  DD  254 
and/or  the  applicable  classification  guides. 

2.  Contractor  has  a  below  standard/ formal  continuing 
Security  Training  program  for  contractor/subcontractor 
personnel . 

3.  Few  Inexperienced  Contractor  Security  personnel  were 
used. 


4.  No  major  findings  In  all  Security  Inspections. 

5.  Documentation  was  periodically  controlled  In  accordance 
with  Industrial  Security  requirements. 

6.  Lines  of  authority  and  responsibility  were  periodically 
established  within  the  security  organization. 

7.  Staffing  was  the  minimum  to  accomplish  the  tasks 
required  by  security  directives  and  the  needs  of  the  program 
In  a  timely  manner. 

8.  Communications  were  minimal  to  facilitate  timely, 
responsive  replies  to  oral  and  written  communications. 

9.  Security  management  staff  does  minimal  planning  ahead 
for  all  security  and  related  program  elements  to  provide 
staff,  material  and  facility  resources  to  accomplish 
projected  requirements. 

10.  Billet  structure  was  minimum  to  meet  current  and 
projected  future  program  requirements. 

11.  The  Contractor  does  the  minimum  In  furnishing 
brlef/debrlef  statements  In  a  timely  manner  and  does  the 
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minimum  to  avoid  double  billeting  by  assuring  that  existing 
personnel  were  debriefed  in  a  timely  manner. 

12.  The  Contractor  does  the  minimum  for  timely  process 
access  requests  to  appropriate  Government  agencies:  does  the 
minimum  to  track  progress  and  report  status  to  cognizant 
Government  program  and  security  management. 

13.  The  Contractor  was  somewhat  timely  in  development  and 
revising  program  unique  security  practices  and  procedures 

( SPP ' s ) ;  adherence  to  approved  SPP ' s . 

14 .  The  Contractor  was  somewhat  timely  in  development  and 
revising  cover  stories  for  all  program  requirements. 

15.  Contractor  does  the  minimum  to  provide  necessary 
support  to  acquire  accreditation  or  approval  of  facilities 
for  special  program  requirements. 

16 .  Contractor  does  the  minimum  to  adhere  to  the  security 
guides  and  systems  of  control  number  assignment  to  produce  a 
system  of  accountability  that  provides  absolute  traceability 
and  minimizes  possible  compromise  of  program  material. 

17.  The  Contractor  makes  an  effort  to  take  all  necessary 
steps  to  eliminate  security  infractions,  conduct  and 
document  investigations  when  necessary,  and  does  not  prevent 
recurrences . 

Very  Good  Rating  f80  -  89  points K  The  contractor  shall 
earn  this  rating  and  associated  points  if  the  majority  of 
the  'Good"  criteria  were  satisfied  plus  the  following: 

1.  Contractor  Security  Program  has  few  deviations  from  the 
Industrial  Security  Program  and/or  the  contract  DD  254 
and/or  the  applicable  classification  guides. 

2.  Contractor  has  a  standard/ formal  continuing  Security 
Training  program  for  contractor/subcontractor  personnel. 

3.  Limited  inexperienced  Contractor  Security  personnel 
were  used. 

4.  Few  findings  in  an  Security  inspection  and/or 
Inspection. 

5.  Documentation  was  controlled  in  accordance  with 
Industrial  Security  requirements. 

6.  Lines  of  authority  and  responsibility  were  established 
within  the  security  organization. 
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7 .  Staffing  was  adequate  to  accomplish  the  tasks  required 
by  security  directives  and  the  needs  of  the  program  In  a 
timely  manner. 

8.  Communications  were  adequate  to  facilitate  timely, 
responsive  replies  to  oral  and  written  communications. 

9.  Security  management  staff  does  adequate  planning  ahead 
for  all  security  and  related  program  elements  to  provide 
staff,  material  and  facility  resources  to  accomplish 
projected  requirements. 

10.  Billet  structure  was  adequate  to  meet  current  and 
projected  future  program  requirements. 

11.  The  Contractor  does  furnishing  brief /debrief  statements 
In  a  timely  manner  and  does  the  avoid  double  billeting  by 
assuring  that  existing  personnel  were  debriefed  In  a  timely 
manner . 

12.  The  Contractor  does  timely  process  access  requests  to 
appropriate  Government  agencies:  does  the  minimum  to  track 
progress  and  report  status  to  cognizant  Government  program 
and  security  management. 

13.  The  Contractor  was  timely  In  development  and  revising 
program  unique  security  practices  and  procedures  (SPP's); 
adherence  to  approved  SPP's. 

14.  The  Contractor  was  timely  In  development  and  revising 
cover  stories  for  all  program  requirements. 

15.  Contractor  does  provide  necessary  support  to  acquire 
accreditation  or  approval  of  facilities  for  special  program 
requirements . 

16 .  Contractor  does  adhere  to  the  security  guides  and 
systems  of  control  number  assignment  to  produce  a  system  of 
accountability  that  provides  absolute  traceability  and 
minimizes  possible  compromise  of  program  material. 

17.  The  Contractor  takes  all  necessary  steps  to  eliminate 
security  Infractions,  conduct  and  document  Investigations 
when  necessary,  and  does  prevent  recurrences. 

Excellent  Rating  (90  -  100  points).  The  contractor  shall 
earn  this  rating  and  associated  points  If  the  majority  of 
the  'Very  Good"  criteria  were  satisfied  plus  the  following: 

1.  Contractor  Security  Progreun  has  no  deviations  from  the 
Industrial  Security  Program  and/or  the  contract  DD  254 
and/or  the  applicable  classification  guides. 
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2.  Contractor  has  a  aggressive  standard/ formal  continuing 
Security  Training  program  for  contractor/ subcontractor 
personnel . 

3.  No  inexperienced  Contractor  Security  personnel  were 
used. 


4.  No  findings  in  an  Security  inspection  and/or 
Inspection. 

5.  Documentation  was  aggressively  controlled  in  accordance 
with  Industrial  Security  requirements. 

6.  Lines  of  authority  and  responsibility  were  clear, 
appropriate  and  established  within  the  security 
organization. 

7.  Staffing  was  above  required  and  appropriate  to 
accomplish  the  tasks  required  by  security  directives  and  the 
needs  of  the  program  in  a  timely  manner. 

8.  Communications  were  excellent  to  facilitate  timely, 
responsive  replies  to  oral  and  written  communications. 

9.  Security  management  staff  does  exceptional  planning 
ahead  for  all  security  and  related  prograun  elements  to 
provide  staff,  material  and  facility  resources  to  accomplish 
projected  requirements. 

10.  Billet  structure  was  appropriate  to  meet  current  and 
projected  future  program  requirements. 

11.  The  Contractor  aggressively  furnishing  brief /debrief 
statements  in  a  timely  mariner  and  aggressively  avoids 
double  billeting  by  assuring  that  existing  personnel  were 
debriefed  in  a  timely  manner. 

12.  The  Contractor  aggressively  processes  access  requests 
to  appropriate  Government  agencies:  does  exceptional 
tracking  of  the  progress  and  report  status  to  cognizant 
Government  program  and  security  management. 

13.  The  Contractor  was  aggressive  in  development  and 
revising  program  unique  security  practices  and  procedures 
( SPP ) ;  adherence  to  approved  SPP ' s . 

14.  The  Contractor  was  aggressive  in  development  and 
revising  cover  stories  for  all  program  requirements. 

15.  Contractor  aggressively  provides  all  necessary  support 
to  acquire  accreditation  or  approval  of  facilities  for 
special  pro  am  requirements. 
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16.  Contractor  aggressively  adheres  to  the  security  guides 
and  systems  of  control  number  assignment  to  produce  a  system 
of  accountability  that  provides  absolute  traceability  and  to 
the  maximum  extent  minimizes  possible  compromise  of  program 
material . 

17.  The  Contractor  goes  beyond  all  necessary  steps  to 
eliminate  security  Infractions,  conduct  and  document 
Investigations  when  necessary,  and  does  prevent  recurrences. 
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over  Letter  to  Government  Interviewee 


DEPARTMENT  OF  THE  AIR  FORCE 

MCAOOUAirrtfIS  AMONAimCAL  SVSTmMS  DIVISION  lA^SC) 
wmol  fT.»ATTSNSON  AM  SONCS  PASS.  OHIO 


cv  fTS  US'  '3^' 

Rtquasc  for  Xntarvlaw 


S««  Dlstrlbatlon 

1.  &«qu«st  aithor  you  or  your  daputy  parclclpata  in  an  Intarriaw  conduetad 
by  Hajor  Mark  T.  Huncar,  a  graduata  atudant  from  tba  Air  Forea  Inacltuta  of 
Tachnology  (AFIT) .  Thla  Intartrlaw  will  ganarata  macarlal  for  an  AFZT 
raaaarch  affort  daalgnad  co  dacarmina  If  an  award  faa  is  a  tool  eha  program 
offica  can  usa  co  laprowa  cha  software  acquisition  process. 

2.  Atcacbmanc  1  is  cha  incarwiaw  guide  Major  Hunter  will  usa  co  conduct 
cha  incarwiaw .  Ha  will  call  your  office  cha  weak  of  20  January  co  schedule 
cha  incarwiaw.  As  part  of  his  research,  ha  will  ask  you  co  cosnanc  on  his 
draft  Parforaance  Ewaluacion  Periods  (Aceh  2)  and  draft  Award  Faa  Criteria 
(Aceh  3).  Tour  eoaBsancs  co  accaehSMnts  2  and  3  are  optional,  but  are 
raqueacad  to  eo8q>laca  the  research  affort.  If  your  schedule  precludes  a 
fomal  incarwiaw,  you  also  hawa  cha  option  of  eomplacing  cha  Incarwiaw 
guide  and  racuming  it  co  Major  Hunter  at  FTC/DXDM  by  16  March  1992. 

3.  Tour  cooperation  and  assiscanea  with  this  effort  is  appraeiatad.  The 
information  obtained  will  prowida  data  for  Major  Hunter's  AFIT  student 
thesis  project.  The  data  is  not  intended  co  assess  organization, 
indiwidual,  or  contractor  performance.  All  responses  will  be  held  in  tha 
strictest  eonfidanca.  No  indiwidual.  program  office,  or  contractor  will  be 
assoeiacad  with  any  of  cha  data,  unless  expressly  desired  by  the  System 


STEHART  E.  CRANSTON  4  Aceh 

Brigadier  General,  (TSAF  1.  Cowammant  Intarwiaw 

Vice  Commander  Guide 

2 .  Draft  Parf omanca 
Ewaluacion  Periods 

3.  Draft  Award  Faa 
Ewaluacion  Criteria 

4.  Distribution  List 


Appendix  D;  Government  Interview  Guide 


GOVERNMENT 
INTERVIEW  GUIDE 


Interview  Date: 
Program  Office: 
Program  Address: 

Phone  No: 

System  Program  Office  Director: 

Deputy  Director: 
Interviewee: 
Request  Anonymity: 
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INTERVIEW  QUESTIONS 


PROGRAM  DIRECTOR  INFORMATION 

(This  section  Is  to  determine  your  system  program  office 
background . ) 

Ola.  When  were  you  assigned  to  your  current  position? 


01b.  At  what  level  are  you  certified  under  the  Acquisition 
Professional  Development  Program? 


02.  Have  you  been  a  System  Program  Office  Director  or 
Deputy  Director  on  any  other  program(s)? 

YES  _  NO  _ 

If  yes,  for  which  program(s).  In  which  posltlon(s),  and 
for  how  long? 


03.  Not  counting  your  current  assignment  as  Program  Office 
Director  or  Deputy  Director,  what  Is  the  total  time  you  were 
assigned  to  a  system  program  offices? 


04.  What  specialty  code(s)  were  you  assigned? 


05.  What  system  program  office  dlrectorate(8)  or  divisions 
did  you  work  for  previously?  (For  example,  engineering, 
logistics,  procurement,  program  control,  operations,  test.) 
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PROGRAM  BACKGROUND 

(This  section  is  to  deteinnlne  experience  with  award  fee.) 


06.  Were  you  part  of  any  precontract  award  process  that 
used  award  fee? 

YES  _  NO  _ 

If  yes,  for  which  program(s),  in  which  position(s)  of  the 
process,  and  how  long  were  you  assigned  to  the  progreun? 


07 .  Does  your  program  office  currently  have  an  award  fee 
contract? 

YES  _  NO  _ 

IF  NO,  SKIP  TO  QUESTION  13. 


08.  What  program  or  portion  of  the  progreun  has  an  award  fee 
plan? 


09.  What  phase  is  the  program  in? 


10.  Is  the  progreun' s  software  development  covered  in  the 
award  fee  plan? 

YES  _  NO  _ 

11.  What  is  the  approximate  dollar  value  of  the  contract? 
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12.  What  percent  (%)  of  the  contract's  award  fee  has  the 
contractor  received  In  the  past? 


13.  What  Is  your  experience  with  award  fee  contracts? 

a .  None 

b .  Some 

c.  Considerable 

d.  Extensive 


14.  What  Is  your  experience  with  software  development? 

a .  None 

b .  Some 

c.  Considerable 

d.  Extensive 


AWARD  FEE  PERSPECTIVE 

(This  section  Is  to  determine  your  viewpoint  on  how  an  award 
fee  should  be  used.) 

15.  What  written  guidance  would  help  In  the  development  of 
award  fee  contracts? 


For  Questions  16  through  23,  how  do  you  think  having 
an  award  fee  would  benefit  (or  hinder)  the  program  and 
software? 


16 .  Software  Development? 
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17.  Software  Development  Cost  Control? 


18.  Software  Delivery  Timeliness? 


19.  Software  Maintenance  Costs? 


20.  Software  Reliability? 


21.  Software  Docximentatlon? 


22.  Software  Testability? 


23.  Software  Quality? 


24.  In  addition  to  award  fee,  what  other  tool  does  a 
program  manager  have  to  control  the  software  development 
process? 


25.  How  has  having  the  award  fee  on  your  program  benefitted 
the  overall  cost,  schedule,  and  performance? 


26a.  Do  you  plan  to  have  an  award  fee  on  future  contracts? 
YES  _  NO  _ 

26b.  Why  or  why  not? 


If  26a  is  NO,  go  to  27. 


26c.  Will  the  award  fee  on  future  contracts  include 
software  development? 

YES  _  NO  _ 
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26b.  Why  or  why  not? 


27a.  Was  having  an  award  fee  on  your  progreun  a  hindrance  to 
the  overall  cost,  schedule,  and  performance? 

YES  _  NO  _ 

27b.  Why  or  why  not? 


27c.  If  the  award  fee  was  a  hindrance,  what  did  you  do  to 
overcome  It? 
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The  remaining  questions  are  optional  but  are  requested  to 
complete  the  research  effort. 


28.  Currently,  the  only  written  guidance  for  Award  Fee 
Plans  is  AFSC  "Award  Fee  in  Systems  Acquisition  -  A  Handbook 
for  Program  Directors  and  Contracting  Officers".  Please 
comment  on  attachments  2  and  3  and  answer  the  following 
questions . 

a.  What  effect  on  the  process  of  software  development  would 
they  have? 

b.  What  benefit  (or  hindrance)  would  they  be  to  the  program 
office  and  the  contractor  if  made  part  of  the  request  for 
proposal  and  source  selection? 

c.  Would  they  help  the  contractor  understand  what  must  be 
accomplished  to  receive  an  award  fee? 

Please  comment  below  or  directly  on  the  attachments. 


If  not  planning  to  have  a  formal  interview,  please  return 
the  guide  and  all  attachments  to: 

FTC/DXDM 

WRI6HT-PATTERS0N  AFB,  OHIO  45433-6508 
ATTN;  MAJOR  MARK  T.  HUNTER 


Note:  Attachments  2  and  3  referenced  above  are  Appendix  A 
and  B  of  this  Thesis. 
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Appendix  E;  Cover  Letter  to  Contractor  Interviewee 


FASTC/SZDM 


DEPARTMENT  OF  THE  AIR  FORCE 
HEADQUARTERS  AIR  FORCE  FOREIGN  TECHNOLOGY  CENTER  (AFIC) 
WRIGHT-PATTERSON  AFB,  OHIO  4S433-650B 


1  7  JAN  199? 


anjiCT  Request  for  Telephone  Interview 


TO; 


1.  Request  either  you  or  your  deputy  pertlclpete  In  en  Interview  to 
generete  meterlel  for  an  Air  Force  Institute  of  Technology  (AFIT)  research 
effort  designed  to  determine  If  an  award  fee  Is  a  tool  the  program  office 
can  use  to  Improve  the  software  acquisition  process.  Your  participation  In 
this  survey  Is  optional. 

2.  Attachment  1  Is  the  Interview  guide  I  will  use  to  conduct  the 
Interview.  As  part  of  my  research,  I'd  like  your  comments  on  the  draft 
Performance  Evaluation  Periods  (Atch  2)  and  draft  Award  Fee  Criteria 
(Atch  3).  Your  comments  to  attachments  2  and  3  are  optional,  but  are 
requested  to  complete  the  research  effort.  If  your  schedule  precludes  a 
formal  Interview,  you  also  have  the  option  of  completing  the  Interview 
guide  and  returning  It  to  me  at  FASTC/DRDM  by  16  March  1992.  If  you  need 
to  contact  me,  I  can  be  reached  at  (513)  257-4166. 

3.  Your  cooperation  and  assistance  with  this  effort  Is  appreciated.  The 
Information  obtained  will  provide  data  for  my  AFIT  masters  thesis  project. 
The  data  Is  not  Intended  to  assess  organisation.  Individual,  or  contractor 
performance.  All  responses  will  be  held  In  the  strictest  confidence.  No 
Individual,  program  office,  or  contractor  will  be  associated  with  any  of 
the  data,  unless  expressly  desired  by  the  System  Program  Office  Director. 

MARK  T.  HUNTER,  MaJ ,  USAF  3  Atch 

Part  Time  AFIT  Student  1.  Contractor  Interview 

Guide 

2 .  Draft  Performance 
Evaluation  Periods 

3.  Draft  Award  Fee 
Evaluation  Criteria 


1st  Ind, 

TO; 

Your  participation  In  Major  Hunter’s  survey  on  award  fee  Is  completely 
optional  and  there  shall  be  no  direct  charge  to  any  contract  as  a  result  of 
you  completing  the  questionnaire  or  Interview  by  the  AFIT  student. 
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Appendix  F;  Contractor  Interview  Guide 


CONTRACTOR 
INTERVIEW  GUIDE 


Interview  Date: 


Contractor: 


Program: 


Program  Address: 


Phone  No: 
Program  Manager: 
Deputy  Program  Manager: 

Interviewer: 
Request  Anonymity: 
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INTERVIEW  QUESTIONS 


PROGRAM  DIRECTOR  INFORMATION 

(This  section  Is  to  determine  your  progreun  office 
background . ) 

Ola.  When  were  you  assigned  to  your  current  position? 


01b.  Have  you  taken  the  Program  Managers  Course  at  the 
Defense  Systems  Management  College? 


02.  Have  you  been  a  Program  Manager  or  Deputy  Program 
Manager  on  any  other  program(s)? 

YES  _  NO  _ 

If  yes,  for  which  program(s).  In  which  posltlon(s),  and 
for  how  long? 


03.  Not  counting  your  current  assignment  as  Program  Manager 
or  Deputy  Program  Manager,  what  Is  the  total  time  you  were 
assigned  a  program? 


04.  What  are  your  specialties,  e.g.  software,  systems 
engineering,  or  mechanical  engineer  etc.? 
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05.  What  parts  of  programs  did  you  work  for  previously? 
(For  example,  engineering,  logistics,  procurement,  program 
control,  operations,  test.) 


PROGRAM  BACKGROUND 

(This  section  Is  to  determine  experience  with  award  fee.) 


06.  Where  you  part  of  the  precontract  award  process  that 
used  award  fee? 

YES  _  NO  _ 

If  yes,  for  which  program(s).  In  which  posltlon(s),  and 
for  how  long? 


07.  Does  your  program  currently  have  an  award  fee  contract? 
YES  _  NO  _ 

IF  NO,  SKIP  TO  QUESTION  13. 

08.  What  program  or  portion  of  the  program  has  an  award  fee 
plan? 


09.  What  phase  la  the  progreun  in? 


10.  Is  the  program's  software  development  covered  In  the 
award  fee  plan? 

YES  _  NO  _ 
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11.  What  Is  the  approximate  value  of  the  contract? 


12.  What  percent  (%)  of  the  contract  award  fee  has  your 
progress  received  in  the  past? 


13.  What  is  your  experience  with  award  fee  contracts? 

a .  None 

b .  Some 

c.  Considerable 

d.  Extensive 


14.  What  is  your  experience  with  software  development? 

a .  None 

b .  Some 

c.  Considerable 

d.  Extensive 


AWARD  FEE  PERSPECTIVE 

(This  section  is  to  determine  your  viewpoint  on  how  an  award 
fee  should  be  used . } 

15.  What  written  guidance  would  help  in  the  development  of 
award  fee  contracts? 


For  Questions  16  through  23,  how  do  you  think  having  an 
award  fee  would  benefit  (or  hinder)  the  program  and 
software? 


16.  Software  Development? 
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17.  Software  Development  Cost  Control? 


18.  Software  Delivery  Timeliness? 


19.  Software  Maintenance  Costs? 


20.  Software  Reliability? 


21.  Software  Documentation? 


22.  Software  Testability? 


23.  Software  Quality? 


24.  In  addition  to  award  fee,  what  other  tool  does  a 
progr2un  manager  have  to  control  the  software  development 
process? 


25.  How  has  having  the  award  fee  on  your  program  benefitted 
the  overall  cost,  schedule,  and  performance? 


26a.  Do  you  plan  to  have  an  award  fee  on  future  contracts? 
YES  _  NO  _ 

26b.  Why  or  why  not? 


If  26a  is  NO,  go  to  27. 
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26c.  Will  the  award  fee  on  future  contracts  Include 
software  development? 

YES  _  NO  _ 

26b.  Why  or  why  not? 


27a.  Was  having  an  award  fee  on  your  progreun  a  hindrance  to 
the  overall  cost,  schedule,  and  performance? 

YES  _  NO  _ 

27b.  Why  or  why  not? 


27c.  If  the  award  fee  was  a  hindrance,  what  did  you  do  to 
overcome  It? 
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The  remaining  questions  are  optional  but  are  requested  to 
complete  the  research  effort. 


28.  Currently,  the  only  written  guidance  for  Award  Fee 
Plans  is  AFSC  "Award  Fee  in  Systems  Acquisition  -  A  Handbook 
for  Program  Directors  and  Contracting  Officers”.  Please 
comment  on  attachments  2  and  3  and  answer  the  following 
questions . 

a.  What  effect  on  the  process  of  software  development  would 
they  have? 

b.  What  benefit  (or  hindrance)  would  they  be  to  the  program 
office  and  the  contractor  if  made  part  of  the  request  for 
proposal  and  source  selection? 

c.  Would  they  help  the  contractor  understand  what  must  be 
accomplished  to  receive  an  award  fee? 

Please  comment  below  or  directly  on  the  attachments. 


If  not  planning  to  have  a  foirmal  interview,  please  return 
the  guide  and  all  attachments  to: 

FTC/DXDM 

WRIGHT-PATTERSON  AFB,  OHIO  45433-6508 
ATTN:  MAJOR  MARK  T.  HUNTER 


Note:  AttachsMnts  2  and  3  referenced  above  are  Appendix  A 
and  B  of  this  Thesis. 
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Memo  from  Ai 
Survey 


JAM  1992 

MEMO  TO:  ASD/CV 

SUBJECT:  AFIT  Thesis  on.  Award  Fee  (Your  Memo,  3  Jan  92) 

1.  We  reviewed  the  paclcage  and  also  discussed  it  with  JAG.  JAG's  opinion  is 
that  the  AFIT  student  should  go  to  the  contractors  directly  for  their  input 
and  not  go  through  the  program  offices.  We  agree  that  going  through  the 
program  offices  to  the  contractors  could  be  interpreted  by  the  contractors  as 
being  an  official  request  tinder  a  contract  and  there  could  be  more  chance  of 
the  contractor  charging  his  time  directly  to  a  contract.  If  the  AFIT  student 
went  directly  to  the  contractor  he  would  not  be  associated  with  any  particular 
program  and  any  charge  the  contractor  might  have  would  be  to  a  general 
overhead  accotmt  which  would  be  spread  over  all  government  contracts. 

However,  even  the  total  amoimt  of  any  charge  would  probably  be  very  minimal. 

2.  If  the  decision  is  made  to  request  input  from  the  contractor  as  the 
student  has  set  up  the  package,  there  should  be  some  minor  changes  to  the 
first  Indorsement  which  the  SPO  personnel  sign.  The  contractor  should  be  told 
that  their  response  is  on  a  voltmtary  basis  and  that  there  should  be  no  direct 
charge  to  any  contract  as  a  result  of  their  completing  the  questionnaire  or 
being  interviewed  by  the  AFIT  student. 

3.  In  another  topic,  one  area  of  the  package  is  somewhat  confusing.  The 
letter  to  the  contractor  says  that  the  student  will  call  the  contractor  to  set 
up  an  Interview.  Paragraph  3  of  the  letter  for  your  signature  to  the  SPO 
Directors  says  that  the  contractor  can  mail  his  responses  back  indicating 
there  would  be  no  interview. 
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Appendix  H;  Government  Interviewee  Responses 


INTERVIEW  QUESTIONS 
PROGRAM  DIRECTOR  INFORMATION 

(This  section  Is  to  determine  your  system  program  office 
background . ) 


Ola.  When  were  you 


a. 

9  months 

b. 

Aug  89 

c. 

15  Apr  90 

d. 

Oct  91 

e. 

3  Sep  91 

f . 

1  1/2  years 

g- 

1988 

h. 

18  mo 

1. 

Aug  91 

j. 

May  91 

k. 

Jul  87 

1. 

Apr  90 

m. 

Jan  89 

n. 

Aug  91 

o. 

Mar  90 

P- 

6  years 

q- 

4  years 

r. 

Mar  89 

s. 

May  91 

t. 

Sep  89 

u. 

Sep  88 

V. 

N/C 

w. 

Jun  90 

assigned  to  your 


current  position? 


01b.  At  what  level  are  you  certified  under  the  Acquisition 
Professional  Development  Progreun? 


a. 

b. 

c. 

d. 

e. 

f . 

g- 

h. 

1. 

j. 

k. 

l. 


PM 

- 

Level 

III 

PM 

- 

Level 

III 

PM 

- 

Level 

III 

Requested,  Test  -  Level 

PM 

- 

Level 

III 

PM 

- 

Level 

HI, 

Cont  -  Level  III 

PM 

- 

Level 

III 

PM 

- 

Level 

III 

PM 

- 

Level 

III 

PM 

- 

Level 

III 

PM 

- 

Level 

III 

Requested 

PM 

- 

Level 

III 

PM 

- 

Level 

III 
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m.  PM  -  Level  I 

n.  should  be  PM-Level  III 

o.  0 

p.  0 

q.  Contracting  -  Level  II  Pre/Post  Award 

r.  PM  -  Level  III 

s.  PM  -  Level  III  Requested 

t.  PM  -  Level  II,  Application  for  Level  III  at  SAF/AQS 

u.  PM  -  Level  III 

V.  N/C 

w.  PM  -  Level  III 


02.  Have  you  been  a  System  Program  Office  Director  or 
Deputy  Director  on  any  other  program(s)? 

a.  no 

b.  no 

c.  no 

d.  yes 

e.  no 

f .  no 

g.  yes 

h.  no 

i.  yes 

j .  no 

k.  no 

l .  yes 

m.  no 

n.  yes 

o.  yes 

p.  no 

q.  no 

r .  no 

s .  yes 

t .  no 

u.  no 

V.  N/C 

w.  yes 

If  yes,  for  which  program(s),  in  which  position(s),  and 
for  how  long? 

a.  N/A 

b.  N/A 

c.  N/A 

d.  TACIT  RAINBOW  -  4  1/2  years 

e.  N/A 

f .  N/A 

g.  Program  Manager  for  B1  Production 

h.  N/A 
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1.  Deputy  Prog  Olr,  Electronic  Combat  &  Reconnaissance 

(ASD/RW)  Jan  89  -  Nov  89,  Dep  PD,  AMRAMM,  Nov  89  - 
Aug  91 

j.  N/A 

k.  N/A 

l.  Deputy  Program  Director  Air  to  Surface  Guided 
Weapons 

m.  N/A 

n.  All  engines 

o.  Strategic  Air  Command  Deputy  Program  Manager  for 
Acquisition  Logistics  B1 

p.  N/A 

q.  N/A 

r.  N/A 

s.  Classified  -  Director  of  two 

t.  N/A 

u.  N/A 

V.  N/C 

w.  Deputy  Small  ICBM  -  11  month 


03.  Not  counting  your  current  assignment  as  Program  Office 
Director  or  Deputy  Director,  what  Is  the  total  time  you  were 
assigned  to  a  system  program  offices? 

a.  0 

b.  15  years 

c .  14  me 

d.  10  1/2  years 

e .  3  years 

f.  19  years 

g.  29  years 

h .  7  years 

1.  10  Years 

j .  13  years 

k.  12  years 

l .  7  years 

m.  0 

n.  30  years 

o .  6  years 

p.  30  years 

q.  l"^  1/2  years 

r.  19  years 

s.  19  years 

t .  4  Years 

u.  10  years 

V.  N/C 

w.  12  years 
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c  rt  a  M  iQ  >0  o  9  8  H>  :ru3  Hi  a  a  n  cr  a 


04.  What  specialty  code(s)  were  you  assigned? 


a.  2716 

b.  27XX 

c.  2916 

d.  27XX,  0029 

e.  29XX,  64XX 

f.  2716,  2724,  2835 

g.  2825,  2716 

h.  2716,  2724 

i.  N/A  Gov  Civil 

j.  2716,  28XX 

k.  0029,  6416 

l.  0029,  2716,  28XX 

m.  2716 

n.  2716 

o.  2806 

p.  28XX 

q.  1102  civilian 

r.  2911,  2916,  2716,  0029 

s.  2716,  2635,  3036 

t.  2855 

u.  2825,2816 

V.  N/C 

w.  5135  (49XX),  2845,  2816,  2716 


05.  What  system  program  office  directorate(s}  or  divisions 
did  you  work  for  previously?  (For  example,  engineering, 
logistics,  procurement,  program  control,  operations,  test.) 

Engineering 
Projects,  Test 
Projects 

Configuration  Management,  Projects 
Configuration  Management,  Data  Management 
Program  Control,  Program  Management 
Flight  Test,  Program  Control 
Projects 
Projects 

Projects,  Engineering,  Test 
Manufacturing,  Quality  Assurance 
Program  Management 
None 

Engineering 
Program  Management 
Engineering 
Contracting 

Test  and  Deployment,  Program  Control,  Engineering, 
Operations 

Director  Space  Weapons,  All 
Integration  and  Operations 
Engineering 


127 


V. 

w. 


N/C 

Engineering,  Program  Management 


PROGRAM  BACKGROUND 

(This  section  Is  to  determine  experience  with  award  fee.) 

06.  Were  you  part  of  any  precontract  award  process  that 
used  award  fee? 


a. 

no 

b. 

yes 

c. 

N/A 

d. 

yes 

e. 

yes 

f . 

yes 

g- 

no 

h. 

no 

1. 

no 

J- 

yes 

k. 

yes 

1. 

no 

m. 

no 

n. 

yes 

o. 

no 

P« 

yes 

q* 

yes 

r . 

yes 

s . 

yes 

t. 

yes 

u. 

no 

V. 

N/C 

w. 

yes 

If  yes,  for  which  progreun(s).  In  which  posltlon(s)  of  the 
process,  and  how  long  were  you  assigned  to  the  progreun? 


a. 

N/A 

b. 

AMRAAM  Launcher  -  Front  office  oversight 

-  3 

years 

c. 

N/A 

d. 

TACIT  RAINBOW  /  Production  non  recurring 

contract 

e. 

ASD  Deputy  Contracting  and  Manufacturing 

-  2 

years 

f . 

TR  1  Airframe  79-81 

g* 

N/A 

h. 

N/A 

1. 

N/A 

j. 

NK  15 

k. 

C 130  Program  Director 

1. 

N/A 

m. 

N/A 

n. 

Aircraft  Engines 

128 


O.  N/A 

p.  First  award  fee  on  VITA 

q.  ADI,  F16 

r.  Classified  Weapon  System/Configuration  Management/4 
years  Vandenberg  AFB  Space  Shuttle  Deployment 
Program/Chlef ,  Business  Management,  Dir  Program 
Offlce/4  years.  Vertical  Launch  Systems /Program 
Manager/ 3  years 

s.  All  levels 

t.  Space  Test  Progreun  P91-1  mission  Source  Selection 

Evaluation  Board  Team  Chief 

u.  N/A 

V.  N/C 

w.  Program  Manager  Small  ICBM  Basing  Director  ASAT 

Integration,  ASAT  IV&V 


07 

.  Does  your  program  office 

currently  have  an  award  fee 

contract? 

a. 

no  (FPIF,  CPIF,  AND 

CPFF  ONLY) 

b. 

yes 

c. 

N/A 

d. 

yes 

e. 

yes 

f. 

yes 

g- 

yes 

h. 

yes 

1. 

no 

j. 

yes 

k. 

yes 

1. 

yes 

m. 

yes 

n. 

yes 

o. 

yes 

P- 

yes 

q* 

yes 

r. 

yes 

s. 

no 

t. 

yes 

u. 

yes 

V. 

yes 

w. 

yes 

IF 

NO,  SKIP  TO  QUESTION  13. 
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08.  What  program  or  portion  of  the  program  has  an  award  fee 
plan? 

a.  The  0053  (FPIF)  contract  has  been  modified  by  adding 

the  Continuing  Software  Improvement  Program  (CSIP) . 
CSIP  Is  an  award  fee  program  focused  on  offering 
Incentives  to  the  contractor  to  Improve  their 
software  development  process.  Hope  to  Initiate 
award  fee  program  by  1  June  92. 

Launcher  Contract 
N/A 

EMD  -  Weapon  System  and  Engine 
Qualification  of  2nd  Source 
Lot  3-4  Production 
Development 
CPAF  Contract 
N/A 

Progreun  Management,  ILS  Management,  Systems 
Engineering  Management,  Production  Planning 
Production 

Basket  SPO  with  20  programs  -  CEDA,  Design  effort 
for  new  fuse  for  25mm  bullet  for  AC130 
CPAF  -  Research  and  Development 
The  total  component  Improvement  program 
All,  Quality  of  technical  work.  Systems  Engineering, 
Program  Management,  Special  Interest 
Hole  program 
Hole  progreuR 

Integrated  Apogee  Boost  Subsystem  (lABS)  Production. 
In  the  past,  our  satellite  production  has  also  had 
award  fee. 

N/A 
All 

Production  and  Launch  Support  Services 
Delta 

React,  AVR 


.  What  phase  Is  the  program  In? 

DT&E  /  Formal  Testing 

Production 

N/A 

EMD 

FSD 

Production 

Concept  Exploration,  EFlll  Engineering  Development 

Final  part  of  Concept  Exploration 

N/A 

FSD 

Production 

EMD 
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m.  Full  Scale  Development /Product Ion 

n.  All  phases 

o.  Concept  Exploration 

p.  Test  and  Evaluation 

q.  Engineering  and  Development /Advanced  Development 

r.  Production  and  Development 

s.  N/A 

t .  Development 

u.  Production 

V.  Production 

w.  FSD 


10.  Is  the  progreun's  software  development  covered  In  the 
award  fee  plan? 

a.  yes 

b.  no 

c.  N/C 

d .  yes 

e.  no 

f .  yes 

g.  yes  (EFlll) 

h.  yes 

1.  N/A 

j.  yes 

k.  no 

l.  no  (AIJL  Hardware) 

m.  yes 

n.  yes 

o.  yes 

p.  yes 

q.  yes 

r.  no  (Not  specifically,  however  overall  program 

management  does  Include  contractor  performance 
Including  software  development) 

s.  yes 

t.  no  (not  as  a  specific  Item) 

u.  no 

V.  N/A 

w.  yes 


11.  What  Is  the  approximate  dollar  value  of  the  contract? 

a.  $940  N 

b.  $72.8  M 

c.  N/A 

d.  $12  B  total  Weapon  System  and  Engine 

e.  $10  M 

f.  $800  N 

g.  $230  M 

h.  $488  M 
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i.  N/A 

j.  $162  M 

k.  $1.2  B 

l.  $1  M 

m.  $150  M 

n.  $49.5  M/$45  M 

o.  $36  M 

p.  $44  M 

q.  $44  M  /  $32  M 

r.  $118  M 

s.  $1.7  B 

t.  $50  M 

u.  $939.7  M 

V.  $850  M 

w.  $45  M  /  $33  M 


12.  What  percent  (%)  of  the  contract's  award  fee  has  the 
contractor  received  in  the  past? 

a.  Between  60  and  75  % 

b.  Period  1-44  %,  Period  2-54%,  Period  3-100% 

c.  N/A 

d.  Less  than  90%  for  period  1  (Need  to  cut  this  data 
by  contract  type) 

e.  N/A 

f.  70  %  3  of  5  Periods  Complete 

g.  Program  Just  Started 

h.  81  -  71  % 

1.  N/A 

j.  50  -  80  % 

k.  80  -  90  % 

l.  No  periods  completed  yet 

m.  70  % 

n.  74-86%  /  75-80% 

o.  45  -  52% 

p.  30  -  60% 

q.  30  -  60%  /  50% 

r.  1988  -  90%,  1989  -  82%,  1990  -  83%,  1991  -  87% 

s .  New  contract 

t.  60  -  95  % 

u .  New  Contract 

V.  30  -  80% 

w.  82%  /  76% 


13.  What  Is  your  experience  with  award  fee  contracts? 

a .  Some 

b.  Considerable 

c.  Considerable 

d .  Some 

e.  Extensive 
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f . 

Considerable 

g- 

Considerable 

h. 

Some 

i. 

Some 

j. 

Considerable 

k. 

Considerable 

1. 

Some 

m. 

Some 

n. 

Extensive 

o. 

Some 

P- 

Some 

q- 

Considerable 

r. 

Considerable 

s . 

Extensive 

t. 

Considerable 

u. 

Considerable 

V. 

Considerable 

w. 

Extensive 

14. 

What  Is  your  experience  with  software  development? 

a. 

Some 

b. 

Considerable 

c. 

Some 

d. 

Extensive 

e. 

Some 

£. 

Some 

g- 

Considerable 

h. 

Some 

i. 

Considerable  (All  Bad) 

j. 

Some 

k. 

Considerable 

1. 

Considerable 

m. 

Some 

n. 

Some 

o. 

Some 

P- 

Considerable 

q- 

None 

r . 

Considerable 

8  . 

Considerable 

t. 

Some 

u. 

none 

V. 

None 

w. 

Extensive 
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AWARD  FEE  PERSPECTIVE 


(This  section  Is  to  determine  your  viewpoint  on  how  an  award 

fee  should  be  used.) 

15.  What  written  guidance  would  help  In  the  development  of 

award  fee  contracts? 

a.  1)  ESD  CC  Policy  on  review  of  award  fee 
Determination,  dated  19  Aug  91 

2)  ESD  PK  Policy  Letter  award  fee  Process 
Excellence,  dated  13  Nov  91 

3)  award  fee  In  Systems  Acquisition,  A  Handbook  for 
Program  Directors  (HQ  AFSC/DCS-PK) 

4)  AFR  and  Supplements,  Part  16.4 

b.  Challenge  Is  In  tailoring  criteria  to  small 
manageable  (small)  number  of  criteria  tied  to 
specific  periods. 

N/A 

Who  did  It  before.  Examples 

Guidance  Booklet  -  examples.  Lessons  Learned. 
Criteria 

f.  Written  down  somewhere  the  Pros  and  Cons  of  award 
fee,  where  do  you  get,  the  benefits,  and  what  are 
the  costs.  There  Is  an  administrative  cost 
associated  with  award  fee.  There  Is  some  benefit 
too.  This  document  should  explain  what  Is  an  award 
fee  Intended  to  do,  what  Is  an  award  fee,  and  how  to 
structure  the  award  fee  Plan  to  do  that  job. 

g.  Think  guidance  Is  all  there  -  Need  training.  No 
GOOD  Education  Program. 

h.  More  extensive  data  base  of  good  award  fee  Plans 
(Few  examples  from  ASD/SSD) .  Level  of  experience  - 
Informal  review  -  advice.  Guidance  Group  -  People 
to  go  to  with  questions 

1.  Little.  We  understand  the  concept.  It  Is  however 

rather  manpower  Intensive  and  an  administrative 
burden  to  properly  execute.  But  effective. 

j.  Handbook  -  Templates  -  How  to  fund/implementation. 
How  to  brief  contractor  on  results. 

k .  None 

l.  Examples  of  award  fee  plans  that  work  -  Successful 
progr£uiis  ranging  from  simple  to  complex.  Reluctance 
from  people  In  trenches  because  of  administration 
work  load.  CEDA  award  fee  requires  additional 
efforts. 

m.  Type  of  contract (s)  to  be  used,  objectives  most 
appropriate  to  award  fee,  suggested  areas  covered, 
suggested  criteria,  administration,  teaching  of 
results  (metrics)  le  did  It  do  what  Intended  to  do. 
Standardization  of  %'  (award  amounts),  applicability 
of  base  fees. 
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n.  Acquisition  Handbook  for  award  fees.  More  of  real 
life  examples  other  hand  AFSC  handbook.  Plan  and 
criteria  are  personalities  Involved.  What  other 
people  grade  and  how  the  grade. 

o.  An  easy  awareness  of  the  tools  to  do  the  job. 
Checklist  for  the  process,  examples  {KEEP  IT  SIMPLE} 

p.  Lessons  learned  In  the  for  of  templates.  Period 
breakout  measures 

q.  FAR  and  FAR  Sup,  Samples  of  what  has  worked  In  the 
past 

r.  If  I  was  In  the  beginning  stages  of  developing  an 
award  fee  contract,  there  Is  some  basic  guidance  I 
would  want  to  see:  a)  The  different  contract  types 
that  award  fee  can  fit  into  (e.g.,  CPAF,  FFP/AF, 
etc. ) 

b)  Some  examples  of  how  award  fee  can  be  applied 
(e.g.,  to  Incentives  certain  parts  of  the  contract 
such  as  schedule.) 

c)  Guidance  on  how  the  award  fee  pool  should  be 
(e.g.,  3%  of  contract  value.) 

s.  SSD/PK  has  guide  Lessons  Learned  -  How  to  get  the 
contractor  to  be  responsive  with  changing  funding  - 
C2une  from  ATF  Award  fee  contract. 

t.  Strawman  criteria,  some  examples  with  good  and  bad 
results  (lessons  learned)  that  can  be  tailored  to 
each  program. 

u.  award  fee  Is  a  good  contract  management  tool.  The 
guidance  we  need  should  specify  which  type  contract 
award  fee  can  be  used. 

V.  N/C 

w.  The  award  fee  process  should  be  the  cornerstone 

of  the  entire  contract  management  process.  Tell  the 
contractor  what  you  expect,  tell  him  how  he  Is  doing 
at  mid  term,  grade  him  at  end  of  period  after 
listening  to  your  story  and  his  story  concerning  how 
he  did  against  expectations  set  at  first  of  period. 
Criteria  should  not  be  too  objective.  FDO  should  be 
able  to  measure  contractors  progress  against  the 
odds. 


For  Questions  16  through  23,  how  do  you  think  having 
an  award  fee  would  benefit  (or  hinder)  the  program  and 
software? 

16 .  Software  DeveloEsnent? 

a.  CSIP  uses  the  SEI  Software  Capability  Evaluation 

(SCE)  as  the  evaluation  tool.  The  SCE  is  designed 
to  motivate  the  Contractor  to  improve  their  software 
development  process.  Changes  in  the  Contractor's 
previous  ad  hoc  development  process  have  already 
been  observed. 
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Significant  help  In  EMD  phase. 

N/A 

-  The  same  comment  for  16  through  23  - 
Benefit  can  not  be  quantified 

-  The  same  comment  for  16  through  23  - 
All  the  questions  on  software  would  receive  the  same 
answers  as  hardware.  There  should  not  be  any 
difference. 

It  would  help,  award  fee  In  an  appropriate  tool  but 
must  have  a  good  software  program  manager  to  use  the 
tool. 

Could  help  how  well  they  do  design/make  assessment 
on  design 

-  The  same  comment  for  16  through  23  - 
Should  help  In  all  areas 
Chose  one  area  and  other  may  suffer 

-  The  seune  comment  for  16  through  23  - 
These  all  could  be  elements  In  an  award  fee  plan  for 
the  specific  period  -  In  and  of  themselves  they  are 
not  of  the  level  to  justify  a  plan.  -  The  Impact 
would  be,  I  feel,  generally  positive.  -  But  the 
degree  of  oversight  to  administer  would  be 
considerable. 

-  The  same  comment  for  16  through  23  - 
Definitely  -  The  weights  for  each  area  will  change 
according  to  phase, 
yes 

Yes  award  fees  can  -  difficult  -  maintaining  balance 
between  hardware  and  software 
Benefit  -  Control  and  management  of  software 
management  Indications 

-  The  same  comment  for  16  through  23  - 
award  fees  Is  a  motivator  for  all  programs 
Can  be  applied  to  software  because  software  Is  easy 
to  change 
yes 
yes 

All  depends  on  program  size,  $  of  award  fee  pool, 
and  administration  burden 

r.  Hinder  -  The  contractor  becomes  focused  on  agreed 
upon  software  requirements.  If  award  fee  were  in 
place,  the  contractor  would  not  be  willing  to  tweak 
mission  requirements  as  needed  In  the  development 
process .  Requirements  change . 

s.  -  The  same  comment  for  16  through  23  - 
Never  been  a  hlnderance.  Is  a  great  benefit 

t.  award  fees  are  not  In  themselves  either  good  or  bad 
/  benefit  or  hinder  the  progreun.  The  criteria  must 
reflect  what  the  Government  Is  looking  for.  If  the 
criteria  are  bad  so  are  all  the  results. 

u.  It  will  benefit  both  the  Government  and  contractor. 
The  contractor  will  Identify  requirements.  He 
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cannot  meet  or  will  have  difficulty  meeting  early  in 
the  development  cycle. 

V.  Yes 

w.  Ideal,  provided  the  criteria  permits  FDO  to  grade 

contractor's  handling  of  both  known  unknowns  and 
unknown  unknowns.  FDO  needs  flexibility. 


17.  Software  Development  Cost  Control? 

a.  It  is  expected  that  low  level  detailed  software 

process  information  the  CSIP  program  will  capture 
and  monitor  will  provide  the  Program  Director  the 
requisite  Insight  into  the  Contractor's  software 
process  to  keep  control  of  the  development  costs. 
Moderate.  Other  techniques  may  be  as  effective. 

N/A 
N/C 
N/C 

Indirect  help 
Not  sure 
N/C 
N/C 
N/C 

No  -  Because  of  documentation 
Not  normally  Incentlvlzed 

Yes  -  Help  in  the  since  of  focusing  management 
attention  on  specific  areas 
Depends  on  type  of  contract,  if  cost  type  of 
contract  award  fee  would  have  to  have  high  pool  to 
incentives  cost  control 
N/C 

yes  -  It  should 
Not  effective  tool 

-  The  same  comment  for  17  through  23  - 
Would  benefit  all 

Benefit  -  The  contractor  managers  would  become  more 
Involved  in  regulating  hours.  More  management 
intervention  would  ultimately  decrease  cost,  by 
streamlining  schedules. 

N/C 
N/C 

If  resource  management  is  an  award  fee  criteria,  the 
contractor  will  control  the  number  of  people  working 
the  program  and  will  clearly  define  tasks  being 
worked . 

V.  Depends  on  contract  type 

w.  Seuae  as  16.  Can  needs  to  be  taken  to  allow  FDO  to 

make  a  holistic  assessment  vs.  a  C/SCSC  objective 
criteria.  If  you  don't  set  up  right  criteria  will 
be  forced  to  double  bang  for  technical  problems  (in 
technical  achievement  criteria)  and  cost  problems 
(in  cost  criteria). 
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18.  Software  Delivery  Timeliness? 

a.  Metrics  have  shown  that  software  process  improvement 

translates  directly  into  productivity  improvement 
and  reductions  in  testing  time. 

Somewhat.  What  is  your  metric? 

N/A 
N/C 
N/C 

Might  help.  This  is  where  the  award  fee  should  be 
focused. 

All  goes  back  to  design  /  Measurable  /  Make 
assessment  on  design 
N/C 
N/C 
N/C 
Yes 

Force  up  front  planning  -  will  not  help  in  plan 
execution 

m.  Good  potential  for  benefit 

n.  N/C 

o.  No  help  -  did  not  make  a  difference 

p.  Not  effective  tool 

q.  N/C 

r.  Hinder/Benefit  -  A  software  delivery  award  fee  may 

force  the  contractor  to  deliver  a  substandard 
product.  It  could  also  force  him  to  work  overtime 
to  meet  schedules.  Software  timeliness  in/ariably 
change . 

s.  N/C 

t.  N/C 

u.  Deliverables  at  time  phased  milestones,  let  you  know 

how  well  the  contractor  is  performing  and  you  will 
have  a  great  deal  of  leverage 

V.  If  issue  is  quality  product  delivered  late  vs  poor 

product  delivered  on  time,  award  fee  could 
incentivlzed  wrong  goal  if  schedule  was  rewarded. 

w.  Same  as  above,  being  able  to  set  new  objectives  for 

each  period  gives  contractor  a  chance  to  start  with 
clean  slate  each  period. 


19.  Software  Maintenance  Costs? 

a.  Metrics  have  also  shown  that  software  process 
improvement  will  Increase  the  early  detection  of 
errors  and  decrease  the  numbers  of  errors  per  KSLOC. 
The  cost  to  correct  a  defect  discovered  early  in  the 
development  phase  is  10-90  times  what  it  would  be  in 
field  operation. 

b.  Not  worth  it  to  me.  What  do  you  mean?  Suppose  in 
EMD. 
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C.  N/A 

d.  N/C 

e.  N/C 

£.  Would  not  be  easy  to  measure.  Collateral  benefit 

g.  Could  have  Impact 

h.  N/C 

i.  N/C 

j.  N/C 

k.  Doubtful 

l.  Would  be  tough  -  outside  design  and  award  fee  period 

m.  This  would  be  difficult  to  measure 

n.  N/C 

o.  No  help  -  did  not  make  a  difference 

p.  Does  not  applied 

q.  N/C 

r.  Benefit  -  Maintenance  schedules  can  easily  conform 

to  award  fee  criteria.  Very  appropriate 

s.  N/C 

t.  N/C 

u.  I  don't  believe  it  would  have  much  of  an  impact 

V.  No  -  Too  far  in  the  future,  award  fee  must  be 

timely  -  instant  gratification  to  work, 
w.  Same  as  above,  FDO  can  put  priority  on  specific 

items  with  contractors  management  that  would 
otherwise  would  not  get  emphasize. 


20.  Software  Reliability? 

a.  CSIP  promotes  early  error  identification,  error 
prevention  and  error  reduction.  It  is  expected  that 
this  will  translate  directly  into  highly  reliable 
system. 

b.  Wery  effective 

c.  N/A 

d.  N/C 

e.  N/C 

f.  Will  not  help.  Will  not  reduce  first  numbers  of 
failures,  indirectly 

g.  Goes  back  to  design 

h.  N/C 

1.  N/C 

j.  N/C 

k.  Doubtful 

l.  Force  upfront  planning  -  Dependent  on  contractor 
culture  -  May  focus  contractor  effort 

m.  Yes,  If  you  get  specific  criteria 

n.  N/C 

o .  No  help 

p.  Can  not  tell 

q.  N/C 
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r.  Benefit  -  Once  the  software  Is  operational, 
reliability  can  easily  be  measured  and  awarded 
appropriately . 

s.  N/C 

t.  N/C 

u.  I  don't  believe  It  would  have  much  of  an  Impact 

V.  No  -  See  19 

w.  Warranties  may  be  better  unless  award  fee  period  Is 

concurrent  with  fielding. 


21.  Software  Documentation? 

a.  CSIP  requires  process  Improvements  In  all  areas  of 
the  development  process.  Because  documentation  Is 
an  Integral  part  of  the  development  process.  It  Is 
expected  that  Improvement  In  the  process  will  result 
In  quality  documentation. 

b.  How  do  you  define? 

c.  N/A 

d.  N/C 

e.  N/C 

f.  Can  be  used  for  leverage.  If  part  of  award  fee 
process . 

Would  help  quality  /  timeliness 
N/C 
N/C 
N/C 
Yes 

Yes  -  can  be  effective  can  see  Into  process 
Very  effective  with  tremendous  payoff  down  stream 
N/C 

Yes  -  small  enough 
Not  effective  tool 
N/C 

Benefit  -  Quality  and  timeliness  of  documentation 
can  be  easily  determined  by  cross  checking  DIDs  and 
program  milestones.  Appropriate. 

s.  N/C 

t.  N/C 

u.  Some  what  of  an  Impact.  This  relates  to 

deliverables.  You  must  specify  what  documentation 
you  want  delivered  and  specify  the  schedule. 

V.  Possibly  -  but  how  do  you  rate  good  vs  better.  Ask 

for  what  you  want  and  insist  on  It  contractually 
without  reward 

w.  -  The  same  comment  for  21  through  23  - 

See  16  through  20 
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22.  Software  Testability? 

a.  CSIP  promotes  the  evolution  of  software  engineering 
from  an  ad  hoc  labor  intensive  activity  to  a 
managed,  quality  controlled  process.  It  is  expected 
that  this  disciplined  approach  will  result  in  a 
quality  test  baseline. 

b.  Probably  less  effective 

c.  N/A 

d.  N/C 

e.  N/C 

f.  Could  If  part  of  award  fee  process 

g.  No  Comment  /  Not  enough  Information  on  these 

h.  N/C 

i.  N/C 

j.  N/C 

k .  Maybe 

l.  No  -  Culture  within  contractor  is  the  driving  force. 
May  help  with  upfront  planning 

m.  Yes,  again  if  you  get  specific  criteria  that  is 
measurable  and  meaningful 

n.  N/C 

o.  Yes  -  The  hold  contract  price 

p.  Not  effective  tool 

q.  N/C 

r.  Benefit  -  award  fee  in  this  area  will  force  the 
contractor  to  use  innovative  and  creative  techniques 
to  formally  test  and  demonstrate  software 
requirements.  Warranted. 

s .  N/C 

t.  N/C 

u .  No  impact 

V.  N/C 

w.  -  The  same  comment  for  21  through  23  - 

See  16  through  20 


23.  Software  Quality? 

a.  CSIP  Incentivlzes  the  Contractor  to  pay  close 
attention  to  his  software  development  process.  It 
is  expected  that  this  close  scrutiny  will  result  in 
process  Improvement  which  in  turn  will  result  in 
product  Improvement.  Continued  process  Improvement 
will  being  the  Contractor  to  a  level  enabling  him  to 
measure  adherence  to  that  process  identify  root 
causes  of  poor  quality,  and  correct  those  causes. 

b.  How  do  you  define? 

c.  N/A 

d.  N/C 

e.  N/C 

f .  Indirectly 
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g.  Should  be  part  of  source  selection  to  weed  out 
contractors  with  poor  software  quality 

h.  N/C 

i.  N/C 

j.  N/C 

ic .  Maybe 

l.  Quality/Reliability  -  Upfront  planning  may  be 

effected  -  May  dlsinstentlvizes  contractor  - 
Contractor  may  not  test  -  Difficult  -  field 
experience  -  Focus  on  upfront  planning  and 
Quality/Reliability  mat  be  a  by-product. 

m.  Yes,  again  if  you  get  specific  criteria  that  is 
measurable  and  meaningful 

n.  N/C 

o.  Can  not  tell  -  Contractor  has  an  approved  software 
development  process 

p.  Not  effective  tool 

q.  N/C 

r.  Benefit  -  Delivered  software  must  conform  to  quality 

standards  spelled  out  in  MIL-STD-.  Inadequate 
software  should  be  penalized. 

s.  N/C 

t.  N/C 

u.  Not  much  of  an  impact.  How  would  you  measure 

software  quality?  What  would  be  your  criteria?  If 
you  mean  performance  then  it  would  have  a  specific 
Impact 

V.  How  do  you  evaluate  -  This  seems  binary  to  me  for 

imbedded  software.  It  work  or  it  don't.  If  it 
don't  its  a  contract  problem. 

w.  -  The  same  comment  for  21  through  23  - 

See  16  through  20 


24.  In  addition  to  award  fee,  what  other  tool  does  a 

program  manager  have  to  control  the  software  development 

process? 

a.  The  Joint  STARS  Program  Office  employs  design 
notebook  "Code  Reviews"  focused  on  the  software 
development  process.  A  close  systematic  study  of 
the  audit  trail  provides  insight  into  the  Internal 
workings  of  the  process. 

b.  CPARS  can  be  very  powerful.  Milestone  billing. 

c.  N/A 

d.  DOD-STD-2167A,  Computer  Resources  Working  Group, 
Software  Development  Design  Reviews,  Software 
Integrate  Program 

e.  Progress  Payments  withholds 

f.  Program  Reviews,  Design  Review,  Contract  Type  are 
all  tools,  award  fee  is  a  hammer  in  the  tool  box. 

g.  C/SCSC,  WBS,  Design  Reviews 
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h.  Depends  on  contract  type.  Reviews,  IV&V,  Good 
Approach,  Good  Specifications,  Draft  RFP's 

i.  Metrics  are  imperative  -  there  not  many  good  ones  - 
no  universal  ones  -  very  contractors  depend  -  We 
have  to  understand  their  process. 

j.  CPARs  that  us  award  fee  evaluation  as  input.  -  TPM 
DSMC  Sys  Book  page  146.  -  C/SCSC  -  System 
Engineering  Master  Schedule  -  Entrance  and  Exit 
Criteria 

k.  Oversight  (Reviews),  Progress  payment.  Set  schedule. 
Use  All 

l.  Metrics  -  caution  -  only  if  contractor  uses  them  to 
manage,  e.g.  Contractor  develops  a  manning  plan 
but  never  refers  back  to  it 

m.  Periodic  status Ing  of  management  indicators.  Source 
selection  criteria  and  SDC/CRs.  2167A  compliance, 
program  manager  training,  solid  engineering  support, 
IV&V  (in  some  cases) 

n.  Long  standing  relationships  with  engineering 
contractors 

o.  Having  enough  educated  manpower 

p.  Regulations,  Specifications,  Don't  have  anything 
else. 

q.  Data  Delivery,  Reviews  and  Approval  -  With  holing 
fee  if  data  not  acceptable  -  Inspections  and 
provisions  to  detezmine  work  acceptability 

r.  -  Software  Development  Plan  (approval  authority) 

-  CFSR/CSSR  (Cost  Report  Data) 

-  STP/STD  approval  (Test  Documentation) 

-  Software  Development  Specifications. 

s.  -  Accreditation  of  Software  Capability,  Do  not  view 

contractor  as  advisory,  -  Goal  is  to  work  with 
Government.  -  Process  team  work. 

t.  The  requirements  process 

u.  N/C 

V.  The  contract  -  specifications  -  schedule  -  testing  - 

IV&V 

w.  The  software  engineering  design,  development  and 

test  process  must  be  aggressively  managed.  This  is 
most  critical.  CPAR  with  Interim  CPAR  might  also 
help. 


25.  How  has  having  the  award  fee  on  your  program  benefitted 
the  overall  cost,  schedule,  and  performance? 

a.  TBD 

b.  Yes,  but  SPO  manpower  investment  is  significant. 

c.  N/A 

d.  It  has  helped  to  get  the  program  this  far 

e.  No 

f.  Contractor  focus  work  on  award  fee  items.  The 

Government  needs  to  look  at  where  it  wants  the 
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contractor  to  focus  and  make  sure  these  are  the 
areas  convened  In  the  award  fee  plan. 

g.  BIA,  concept  good,  people  changed  in  the  (EFlll/not 
yet)  middle  so  the  report  card  on  contractor  was 
lacking  all  most  a  year  between  award  fee  Periods. 
Next  award  fee  period  lack  documentation  -  process 
was  lacking.  Need  score  card  on  smaller  periods. 
Develop  tracking  process.  Get  together  with 
evaluators  to  compare  notes.  —  TRAINING  — 

h.  When  you  Incentives  one  thing  It  gets  done  -  Overall 
Management  -  award  fee  was  Best  Tool 

1.  I've  seen  It  on  limited  applications  -  finite  tasks 

-  On  a  macro  basis  -  I  am  very  skeptical  - 

j .  Made  the  program  more  aware  of  what  Is  going  on  - 
Need  a  structure  to  support  Government  data 
collection  for  award  fee.  Need  more  manning.  - 
There  Is  a  return  on  the  Investment  for  award  fee. 

k.  Cost  -  No,  Schedule  -  No,  Contractor  Performance  - 
YES 

l.  TBD,  On  CEOA  focus  management  attention  e.g 
Government  wanted  to  reduce  CEDA  contractor  overhead 
and  the  award  fee  process  made  the  contractor  focus 
management  attention  on  It. 

m.  Only  one  award  fee  period,  most  areas  haven't 
significantly  changed  since  1st  period,  but  I  do 
feel  1st  period  would  have  been  worse  If  not  for 
award  fee.  If  nothing  else.  It  forces  the 
Government  to  give  feedback.  If  administered  right 
and  fairly. 

n.  All  Important  things  get  highlighted  -  it  Is  working 
today  -  Contractor  Is  not  surprised.  Contractor 
looks  at  themselves  during  period 

o.  It  has  not  $4  M  cost  overrun  and  6  month  schedule 
slip. 

p.  It  has  not  been  effective.  May  be  more  effective 
than  Fixed  fee 
It  has  not  made  a  difference 
Our  contractor  responds  to  award  fee  Inputs 
constructively.  lABS  production  Is  50%  complete  and 
projecting  an  underrun.  The  effort  Is  on  schedule 
or  exceeding  It  and  performance  In  the  field  has 
been  demonstrated  and  satisfactorily  proven.  The 
award  fee  has  made  the  contractor  more  responsive  to 
the  SPO  In  some  of  the  more  intangible  areas.  They 
seem  to  listen  to  us  better  when  we  "discuss" 
technical  solutions  to  problems  that  have  arisen. 
Also,  award  fee  seems  to  motivate  them  to  move  out 
Independently  on  problem  ares,  without  our  prodding. 
This  Is  a  subjective  evaluation  of  award  fee,  we 
have  not  done  any  type  of  cost/benefit  analysis. 

s.  Requires  effort  In  administration  and  upfront.  Can 

slow  down  RFP  process  -  With  award  fee,  never  had  a 
project  come  In  late  or  over  cost.  Forces  you  to  do 
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It  better.  Good  managers  will  manage  well.  -  Helps 
managers  with  hammer.  Forces  you  to  justify  your 
position  either  good  or  bad.  Questions  get  asked: 
Did  It  complete  on  time  and  schedule.  Past 
experience.  Fix  Price  overrun.  If  Incentive  fee 
used  on  cost  but  less  product. 

t.  Provides  contractor  Incentive  to  meet  our  program 
goals 

u.  It  has  helped  to  control  cost.  Improve  Government  / 
Contractor  relations  and  contractor  responsiveness 
In  resolving  problems 

V.  We  Incentives  performance  with  a  separate  Incentive, 

award  fee  looks  at  management,  subcontracts 
management,  responsiveness  to  government  and 
Innovation.  It  gets  the  contractors  attention  - 
especially  on  small  Issues. 

w.  Award  fee  Is  absolutely  the  most  powerful  tool 

available  to  program  managers  on  FSD  contracts. 


26a.  Do  you  plan  to  have  an  award  fee  on  future  contracts? 


a. 

b. 

c. 

d. 

e. 

f . 

g- 

h. 

1. 

j. 

k. 

l. 

m. 

n. 

o. 
P- 

q- 

r. 

s. 

t. 

u. 
V. 
w. 


yes 

yes 

yes 

yes 

yes 

yes 

yes 

yes 

It  depends  If  applicable 

yes 

no 

yes 

yes 

yes 

no 

no 

yes 

yes 

yes 

yes 

yes 

yes 

yes 


145 


hiQ  *0  0  3  8  •- >r  t-fc  H- ET  to  Mio  a  n  cr » 


26b.  Why  or  why  not? 

Incentlvlzes  Contractor 
N/C 
N/C 

With  cost  contracts  -  It  Is  a  tool  for  high 
visibility 
Leverage  on  Managers 

1)  Will  allow  better  control  of  cost  and  schedule. 

2)  Work  trades  on  the  margin  of  the  contract. 

In  Cost  Plus  contracts  need  to  find  a  way  to  control 
contractor  costs.  Need  for  evaluation  of 
integration  of  subcontractors  -  not  easy  to 
quantify.  The  ability  to  interface  with  platform 
contractor 

KEY  AREAS  Management,  Engineering,  Schedule,  Strong 
Subcontract  Management 

Maybe  because  the  program  will  drive  requirements 
Depends  totally  on  the  circumstances 
Big  tool  to  motivate  contractor  -  no  other  tool  to 
replace  it. 

To  much  administration  to  be  done 

Because  it  is  an  effective  tool  to  focus  contractor 
and  emphasize  ares  where  the  Government  want 
attention  -  Flexible,  can  change  areas  of  emphases 
based  on  Development/Production  process  -  Incentives 
are  fixed  and  susceptible  to  gaming  by  contractor 
Appears  to  be  a  good  return  on  investment.  However, 
do  not  underestimate  time  and  effort  to  administer 
correctly. 

Gives  the  individual  program  manager  the  authority 
to  make  things  happen  -  A  lot  of  leverage 
The  process  used  to  get  to  the  FDO  has  too  many 
layers.  The  fee  is  determined  by  someone  to  far 
removed  from  the  process.  The  contracting  process 
is  flawed. 

Given  the  effectiveness 
~  The  hassle  is  not  worth  the  effort. 

-  To  much  Government  overhead. 

Under  proper  circumstances 

award  fee  is  the  most  responsive  and  flexible  tool 
we've  used  to  incentives  positive  contractor 
performance . 

s.  It  works  and  Division  Commander  says  so. 

t.  N/C 

u.  It's  a  good  management  tool. 

V.  It  works  to  motivate  contractor  and  formalize  an 

evaluation  process.  It  is  a  "report  card"  for  the 
contractor's  program  suinager. 
w.  I  try  never  to  award  an  FSD  contract  without  it. 


If  26a  is  NO,  go  to  27. 
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26c.  Will  the  award  fee  on  future  contracts  Include 
software  development? 

a.  yes 

b.  yes 

c.  yes 

d.  yes 

e.  yes 

f .  yes 

g.  yes 

h.  yes 

1 .  It  depends 

j.  yes 

k.  no 

l.  yes 

m.  Can  only  speak  to  current  contract 

n.  yes 

o.  no 

p.  no  award  fee  contracts 

q.  yes 

r.  yes 

s.  yes 

t.  no 

u.  no 

V.  probably 

w.  yes 


26d.  Why  or  why  not? 

a.  N/C 

b.  If  phase  of  program  Is  appropriate.  Wouldn't  make 

as  much  sense  In  full  rate  production. 

c.  N/A 

d.  Depends  on  specific  program  requirements 

e.  Important  element  of  program 

f.  It  depends  on  program.  If  software  was  a  major  Item 
It  would  be  used. 

g.  award  fee  has  more  leverage  than  award  fee  Amount 
Incentive  fee's  seem  to  get  less  attention 
Incentlvlzes  Contractor  deliver  on  time  a  quality 
baseline. 

h.  Importance  of  Program  Requirements 

1.  Not  likely  beyond  being  one  of  the  elements  of  the 

larger  picture 

j.  Don't  know  how  else  to  control  without  this  useful 
tool 

k.  To  much  administration  to  be  done 

l.  Because  It  Is  an  activity  who's  track  record  Is 
dismal  -  Area  that  Is  accentual,  software  has  to  be 
there  Is  the  system  Is  to  work  -  It  Is  a  better  way 
to  get  a  contractor's  attention. 

m.  N/C 


n. 

o. 
P- 

q* 

r. 


s. 

t. 

u. 

V. 


w. 


Gives  the  individual  program  manager  the  authority 
to  make  things  happen  -  A  lot  of  leverage 
To  much  work  with  little  or  no  payoff 
It  has  not  been  effective  in  motivating  contractor 
performance . 

Under  proper  circumstances 

The  next  major  software  development  will  be  a  Cost 
Plus  award  fee  (CPAF) .  Cost  Plus  puts  the  risk  on 
the  Government.  The  award  will  make  the  software 
develoEMnent  managers  responsive  to  procurement 
agency  Inputs  and  concerns. 

It  works  and  Division  Commander  says  so. 

Generally  don't  manage  our  contracts  at  that  level 
No  further  software  development  is  planned 
If  we  have  any.  Need  to  consider  the  entire 
contract  incentive  package  and  evaluate  those  things 
that  are  Important  to  the  Government 
N/C 


27a.  Was  having  an  award  fee  on  your  progreua  a  hindrance  to 
the  overall  cost,  schedule,  and  performance? 


a. 

no 

b. 

no 

c. 

no 

d. 

•no 

e. 

no 

f. 

no 

g* 

no 

h. 

no 

1. 

no 

j. 

no 

k. 

no 

1. 

no 

m. 

no 

n. 

no 

o. 

no 

P- 

yes 

q- 

no 

r. 

no 

s. 

no 

t. 

no 

u. 

no 

V. 

no 

w. 

no 
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27b.  Why  or  why  not? 

a.  N/C 

b.  Only  problem  was  spo  manpower  loading. 

c.  N/A 

d.  Make  sure  the  contractor  does  not  "fill  squares." 
Look  at  total  program/product.  Need  to  look  at 
quality. 

e.  Not  significant  enough  dollars 

£.  Focused  contractor  on  areas  the  Government  wanted  In 

this  case  reliability. 

g.  Worked  with  TR  1  progr2un  In  bring  out  manufacturing 
process  from  black  to  white  world. 

h.  Been  an  effective  tool  and  been  able  to  limit  the 
administration  of  award  fee.  Been  a  positive 
Influence 

1.  Got  the  contractor's  attention 

j .  Gave  ability  to  monitor  contractor  and  structure 
Government  Program  Office. 

k.  Did  not  help 

l.  May  be  more  upfront  effort  but  small. 

m.  As  I  said  previously,  their  performance  was  not 
demonstrably  Influenced  by  award  fee  (although  too 
short  to  really  tell).  But,  I  believe  their  Initial 
(first  period)  performance  would  have  been  worse 
without  award  fee. 

n.  N/C 

o.  Didn't  make  a  difference 

p.  Cost  the  Government  time  but  has  not  effected 

contractor  work 

q.  More  Pain  Than  Gain 

r.  award  fee  was  not  a  hindrance  because  It  was  based 
upon  criteria  which  required  the  contractor  to 
exceed  cost,  schedule,  and  performance  requirements 
before  fee  would  be  awarded.  Average  performance 
would  result  In  no  award  fee. 

s.  It  gets  time  In  administration  but  seune  as  Defense 
Acquisition  Board. 

t.  Overall  cost,  schedule,  performance  Is  Included  In 
the  criteria. 

u.  An  award  fee  has  the  contractor  motivated  to  balance 

the  various  aspects. 

V .  It  works 

w.  N/C 


27c.  If  the  award  fee  was  a  hindrance,  what  did  you  do  to 
overcome  It? 

a.  N/C 

b.  N/C 

c.  N/C 

d.  N/C 
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e. 

N/C 

f . 

N/C 

g- 

N/C 

h. 

N/C 

i. 

N/C 

j. 

N/C 

k. 

N/C 

1. 

N/C 

m. 

It  was/ls  a  pain  to  administer  (for  Government)  but 
not  a  hindrance  to  contractor 

n. 

N/C 

o. 

N/C 

P- 

N/C 

q- 

N/C 

r. 

N/C 

s . 

N/C 

t. 

N/C 

u. 

N/C 

V. 

N/C 

w. 

N/C 

The  remaining  questions  are  optional  but  are  requested  to 
complete  the  research  effort. 

28. 

Currently,  the  only  written  guidance  for  award  fee 

Plans  Is  AFSC  "award  fee  In  Systems  Acquisition  -  A  Handbook 

for 

Program  Directors  and  Contracting  Officers".  Please 

comment  on  attachments  2  and  3  and  answer  the  following 
questions . 

28a 

.  What  effect  on  the  process  of  software  development 

would  they  have? 

a. 

N/C 

b. 

N/C 

c. 

N/C 

d. 

N/C 

e. 

Depends  on  emphases  placed  In  award  fee  Plan. 

f . 

Help  put  rigger  In  what  should  have  been  there  In 
the  first  place. 

4 

g- 

N/C 

h. 

Positive,  better  examples,  the  better  yours  will  be 
done,  would  be  a  great  help. 

- 

1. 

Limited  -  very  general. 

j. 

Positive  effect  on  process. 

k. 

Could  not  hurt. 

1. 

Yes  -  Shows /communicates  what  Is  Important  and  how 
to  get  the  pot  of  gold  -  What  he  does  not  do  will 
get  him  zero. 

m. 

N/C 

» 
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n.  Yes  /  CIP  did  have  a  PCO  100  and  award  fee. 

o.  Should  help  process  -  give  more  tools. 

p.  Would  help  -  Liked  criteria. 

g.  Potential  of  Insuring  award  fees  are  done  right  the 

first  time. 

r.  Award  fee  criteria  should  be  less  constricting  up 
until  COR  completion.  After  COR,  award  fee  will 
become  more  beneficial  and  can  be  more  specific. 

s.  Help  as  more  guidance  in  the  field,  it  is  event 
driven . 

t.  N/C 

u.  N/C 

V.  N/C 

w.  N/C 


28b.  What  benefit  (or  hindrance)  would  they  be  to  the 

program  office  and  the  contractor  if  made  part  of  the 

request  for  proposal  and  source  selection? 

a.  N/C 

b.  N/C 

c.  N/C 

d.  N/C 

e.  Benefit  if  acknowledged  upfront  and  agreed  to  as 
part  of  contractor  selection. 

f.  ■  Earlier  is  better  help  focus  Contractor/Government 

on  Contractor  strength/weakness. 

g.  N/C 

h.  Make  more  clear  to  contractor  what  is  required. 

i.  None  -  confusion. 

j.  Excellent  -  1)  Allows  Contractor  to  tell  the 

Government  what  is  stupid.  -  2)  Tells  Contractor 
what  is  expected  and  allows  for  up  front  work  by  the 
Contractor.  ~  3)  Up  front  communication  is 
always  is  good. 

k.  Very  beneficial. 

l.  Yes  -  He  (the  contractor)  would  understand  what  was 
Important  -  better  proposal  -  The  ATF  approach  have 
contractor  write  Statement  of  Work.  Integrated 
management  schedule  Force  contractor  to  think 
through  process  and  think  through  entrance  and  exit 
criteria  for  each  performance  period. 

m.  N/C 

n.  Helpful. 

o.  Great  benefit  to  contract  to  help  him  BID  the  work. 

More  honest  BID. 

p.  Great  help. 

q.  Many  make  plan  negotiable  -  More  information  in  RFP 
the  better  the  proposal. 

r.  A  definite  benefit  to  include  them  as  part  of  the 

RFP  package.  The  contractor  will  probably  be 
receptive  to  almost  ant  type  of  award  fee  plan,  but 
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he  will  want  It  to  be  something  that  Is  achievable. 
An  award  fee  with  criteria  that  are  Impossible  to 
meet  does  nothing  for  either  party. 

s.  Yes,  benefit  to  have  upfront  communications. 

t.  N/C 

u.  N/C 

V.  N/C 

w.  N/C 


28c.  Would  they  help  the  contractor  understand  what  must  be 
accomplished  to  receive  an  award  fee? 

N/C 
N/C 
N/C 
N/C 

Absolutely! ! 

Yes,  but  the  Contractor  will  focus  his  effort  on  the 
award  fee. 

N/C 

Yes,  would  help  contractor  see  what  Is  expected. 

No  -  An  award  fee  plan  Is  by  Its  nature  Is  tailored 
to  the  specific  task  /  contract  /  program  /  phase. 
These  may  be  useful  as  lists  but  they  sure  should 
not  be  dignified  as  'policy'  or  regulation. 

Yes. 

Yes  -  Be  careful  -  Must  be  tailored  -  Need  for 
agreement  on  what  Is  Incentlvlzed  between  Government 
and  Contractor.  Move  Item  13  page  5  to  management. 
Yes  -  Award  fee  will  force  proper  upfront  planning 
and  tracking  to  the  plan.  In  the  past,  progreuns 
have  had  poor  upfront  planning  and  the  contractor 
"pencil  wipes  the  plan  by  the  seat  of  the  pants. 
Communications  between  the  Government  and  Contractor 
Is  key  to  a  successful  program  and  award  fee  will 
Incentive  this  process. 

N/C 

It  would  clarify  what  would  be  required.  Anything 
In  writing  will  help  clarify  requirement  -  6  month 
period  with  3  month  status  reports. 

Yes 
Yes 
Yes. 

Possibly.  Our  experience  with  our  contractor 
revealed  that  when  we  change  to  award  fee  criteria 
similar  to  this  It  caused  great  confusion  on  their 
part.  They  saw  the  criteria  as  a  checklist,  that 
each  one  had  to  be  met  before  moving  on  to  the  next 
section.  Rather,  our  Intent  was  to  take  observed 
behavior  and  see  where  It  fell  In  the  rating. 

Yes 
N/C 
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U.  N/C 

V.  N/C 

w.  N/C 


Final  Comments  made  by  interviewees 

N/C 

What  defines  successful?  How  are  you  tracking 
software?  If  LOC,  can  be  misleading.  Need  to  track 
completed  modules,  etc.  Criteria  is  way  too  broad, 
cannot  watch  ever^hing  —  need  to  select  few 
critical  elements  and  focus  award  fee  on  that.  (Not 
an  Acquisition  checklist) 

N/C 

Need  to  tie  award  fee  periods  to  calendar  not 
milestones.  This  will  keep  the  contractor  for 
focusing  on  near  term  goals  at  the  expense  for  long 
term  program  requirements. 

The  award  fee  plan  needs  to  focus  on  the  product  and 
process.  It  also  must  look  at  the  attitudes  the 
process  will  foster. 

The  F22  award  fee  Plan  is  the  preferred  approach  to 
award  fee  contracts. 

N/C 

Award  fee  will  not  solve  a  bad  SOW  or  Contract. 

Award  fee  Contracts  should  be  managed  by  experienced 
personnel.  Criteria  need  to  hook  back  to  SOW. 

Look  at  growth  reference  to  make  sure  it  is  sending 
the  right  message.  Include  other  category  that 
cover  areas  important  to  program  office. 

Option  -  Need  to  have  a  period  to  tie  to 
milestones. 

Concerns  -  FOT&E  is  not  available  in  all 
acquisitions. 

Final  Comment  -  There  is  a  need  for  training  on 
award  fees  for  0-6  and  Above  as  well  as  below. 

N/C 
N/C 
N/C 

-  Lower  FDO  to  2  Ltr  Program  Director.  Progreun 
Director  has  responsibility  for  program  and  should 
have  FDO  authority. 

l.  N/C 

m.  "Successful  completion"  is  subjective  because  no  PDR 
or  COR  is  100%,  how  does  contractor  know  what  to 
shoot  for  until  afterwards.  How  about  the 
actions/efforts  required  prior  to  PDR/CDR  to  assure 
successful  completion?  -  It  takes  much  more  than 
milestone  completion  (subjective  at  lest)  to  keep 
program  on  track.  This  only  looks  at  an  Important 
but  not  total  part  of  the  picture.  What  about 
quality,  responsiveness,  attitude,  etc.  -  Just 
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delivering  what  the  contract  requires  Is  limited 
thinking,  because  the  contract  cannot  cover  all 
aspects,  requirements,  user  needs,  and  quality  of  a 
product /service.  What  about  user  Inputs? 

n.  N/C 

o.  -  Working  relationship  between  Government  and 

contractor  Is  more  Important  than  award  fee. 

p.  -  Need  to  have  a  lessons  learned  data  base  - 
Streamline  award  fee  process  -  FDO  changed  to  many 
times  -  no  continuity  -  For  this  contract  contractor 
was  not  Interested  In  build  system  but  reluctantly 
agreed.  Because  this  was  a  large  contractor,  the 
award  fee  hammer  was  not  big  enough.  This  was  a 
one-of-a-kind  system  and  the  contractor  didn't  think 
It  was  Important. 

N/C 
N/C 

-  Suggest  using  ATF  approach.  Government  define 
SDR/Ml les tone  I I /Demonstration  Validation. 

Contractor  will  supply  schedule  and  plan  which  will 
be  put  on  contract.  -  Can  help  contractor  with 
capitalization  by  giving  him  award  fee  upfront.  If 
100%  fee  not  earned  must  pay  Government  back. 
Hlnderance  -  Drives  PCO  Crazy.  Award  fee  is  not 

easy  for  Government  but  can  have  great  reward  for 
Government . 

t.  What  ever  guidance  Is  provided,  it  must  be  able  to 
be  tailored  to  meet  the  needs  of  the  program  It  Is 
being  applied  to. 

u.  N/C 

V.  I  have  a  basic  problem  with  their  approach.  In  my 

opinion  the  focus  Is  wrong.  It  should  be  on  what 
the  contractor  can  do  to  earn  the  award  fee,  not 
what  are  the  detractors  that  keep  the  contractor 
from  getting  the  award  fee.  It  should  be  geared 
towards  a  positive  approach,  not  a  negative  one.  In 
other  words,  don't  subtract  for  bad  performance, 
reward  good  performance.  I  strongly  believe  that 
meeting  the  contract  requirements  should  earn  the 
contractor  zero  award  fee.  Exceeding  them  Is  a 
reason  for  rewarding  him  with  fee.  (Depending  on 
the  type  of  contract.  If  award  fee  Is  the  only 
profit  opportunity,  then  he  should  get  50%  for 
success  and  earn  the  rest.)  The  evaluation  criteria 
refers  to  Unacceptable  and  still  provides  up  to  59 
points.  I  can't  believe  that  we  would  give 
anything  for  unacceptable  performance.  Good  on 
their  scale  Is  60  -  79  points.  I  read  this  as  met 
standard.  This  Is  where  you  should  go  from  0-50  for 
how  much  he  exceeded  standard.  Very  good  on  their 
scale  Is  80-90  points  and  excellent  Is  90-100 
points.  On  this  scale  you  would  give  over  half  your 
points  for  unacceptable  and  only  10  points  for  very 


154 


w. 


good  or  excellent.  The  contract  should  specify  firm 
requirements.  If  those  are  not  met,  contractual 
remedies  are  in  order.  Use  award  fee  as  a  carrot  to 
encourage  exceeding  requirements. 

Too  many  categories  and  too  specific.  Traps  FDO. 
Makes  evaluation  too  fragmented.  Criteria  should 
permit  subjective  holistic  judgmental  by  FDO.  See 
our  examples  -  administration  of  process  must  have 
importance  and  credibility  with  contractor. 
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Appendix  I;  Contractor  Interviewee  Responses 


INTERVIEW  QUESTIONS 
PROGRAM  DIRECTOR  INFORMATION 

(This  section  Is  to  determine  your  pro9r2un  office 
background . ) 

Ola.  When  were  you  assigned  to  your  current  position? 

a.  Nov  87 

b.  1988 

c.  1989 

d.  Feb  91 


01b.  Have  you  taken  the  Program  Managers  Course  at  the 
Defense  Systems  Management  College? 

a.  no 

b.  no 

c.  (Progr^un  Manager  Pratt  &  Whitney  course) 

d.  no 


02.  Have  you  been  a  Program  Manager  or  Deputy  Program 
Manager  on  any  other  program(s)? 

a.  yes 

b.  no 

c.  yes 

d.  yes 


If  yes,  for  which  program(s).  In  which  posltlon(s),  and 
for  how  long? 

a.  Advanced  Tactical  Transports  -  2  years  -  Progreun 
Director 

b .  none 

c.  ATF  Dem  Val 

d.  Six  other  programs  as  progreun  manager 
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03.  Not  counting  your  current  assignment  as  Program  Manager 
or  Deputy  Program  Manager,  what  is  the  total  time  you  were 
assigned  a  program? 

a .  2  years 

b.  N/C 

c.  1988 

d .  6  years 


04.  What  are  your  specialties,  e.g.  software,  systems 
engineering,  or  mechanical  engineer  etc.? 

a.  Systems  Engineering,  Operations  Research 

b.  Control  Systems  Engineering 

c .  Engineering/Aero 

d.  Systems  Engineering 


05.  What  parts  of  progreuns  did  you  work  for  previously? 
(For  exeunple,  engineering,  logistics,  procurement,  program 
control ,  operations ,  test . ) 

a .  Engineering 

b .  Engineering 

c .  Engineering 

d.  Hardware  -  manufacturing  Including  procurement 
Production/assembly/ test 


PROGRAM  BACKGROUND 

(This  section  is  to  determine  experience  with  award  fee.) 


06.  Where  you  part  of  the  precontract  award  process  that 
used  award  fee? 


a. 

yes 

b. 

yes 

c. 

yes 

d. 

yes 

If  yes,  for  which  program(8),  in  which  posltion(s),  and 
for  how  long? 

a.  F22 

b.  AFIT/F16  DFCS/AMAS,  Element  Manager/  Chief  Engineer, 

8  years 

c.  F22  engine 

d.  Two  programs  as  program  manager  for  two  years  each 

program 
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07.  Does  your  progreun  currently  have  an  award  fee  contract? 


a.  yes 

b.  yes 

c.  yes 

d.  yes 


IF  NO,  SKIP  TO  QUESTION  13. 


08.  What  progreun  or  portion  of  the  progreun  has  an  award  fee 
plan? 

a.  Total  EMD  Program 

b.  Total  AFTI/F16  CAS 

c.  Total  EMD  Progreun 

d.  Total  program 


09.  What  phase  Is  the  program  In? 

a.  EMD 

b.  Flight  Test 

c.  EMD 

d.  detail  design  of  software 


10.  Is  the  program's  software  development  covered  In  the 
award  fee  plan? 

a.  yes 

b.  yes 

c.  yes 

d.  yes 


11.  What  Is  the  approximate  value  of  the  contract? 

a.  $  65  +  B  (EMD  and  Production) 

b.  $35  M 

c.  $1  1/2  B 

d.  $30  M 


12.  What  percent  (%)  of  the  contract  award  fee  has  your 
prograun  received  In  the  past? 

a.  93% 

b.  50% 

c.  97%  expect  over  90% 

d.  90%  of  each  award  fee  period 
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13. 

What  Is  your  experience  with 

award  fee  contracts? 

a. 

Some 

b. 

Some 

c. 

Some 

d. 

Considerable 

14. 

What  Is  your  experience  with 

software  development? 

a. 

Some 

b. 

Considerable 

c. 

Some 

d. 

Some 

AWARD  FEE  PERSPECTIVE 

(This  section  Is  to  determine  your  viewpoint  on  how  an  award 
fee  should  be  used.) 


15.  What  written  guidance  would  help  In  the  development  of 
award  fee  contracts? 


a. 


b. 


c. 


d. 


Definitive  Instructions  on  how  the  award  fee  Is  to 
be  used,  areas  of  evaluation,  rating  definitions, 
award  fee  process. 

Explanation  of  each  criteria  and  weighing  factor. 
Make  criteria  "end  Item"  not  "process"  oriented. 

Any  criteria  Item  must  be  specifically  Identified  In 
contract . 

1)  Clear  understanding  of  requirements. 

2)  Comprehensive  plan  to  be  judged  by. 

3)  Commitment  by  both  parties  to  Implement  that 
plan. 

4)  Current  plan  well  laid  out  /  good 
objective/subjective  requirements . 

5)  Award  fee  short  course  may  be  a  good  Idea. 
Clearer  definition  of  positive  and  negative  points 
of  customer  that  Imp [act  award  fee  assessment 
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For  Questions  16  through  23/  how  do  you  think  having  an 

award  fee  would  benefit  (or  hinder)  the  program  and 

software? 

16 .  Software  Development? 

a.  yes 

b.  Favorable  impact  -  Put  award  fee  criteria  on  end 

product  and  overall  schedule. 

c.  For  engines  no  effect 

d.  -  Same  comment  for  16  -  23  - 

In  all  the  programs  that  I  have  managed/  maximizing 
our  fee  has  never  served  as  the  primary  motivator.  I 
expect  myself  and  progr2un  personnel  to  put  (?)  the 
highest  quality  product  that  can  be  produced.  By 
"highest  quality  product"  I  am  including  everything  from 
management/  software,  cost  control,  reliability, 
hardware,  test,  etc.  By  stressing  this  approach  we  have 
always  equaled  or  exceeded  the  project  goal  or  award  fee 
goal  that  was  established.  The  amount  of  dollars 
associated  with  the  award  or  profit  had  minor  Influence 
on  performance.  However,  the  review  process  by  which  we 
asses  ourselves  and,  of  course,  the  customer's  assessment 
were  the  primary  cudos  that  motivated  us  and  felt  the 
praise  of  our  efforts.  Changes  required  due  to  short 
comings  were  immediately  addressed  and  perused  vigorously 
to  conclusion  thereby  again  improving  the  quality  of  our 
product . 

So  to  say  that  award  fee  benefit  or  hinder  my 
program,  the  answer  is  no.  However,  the  evaluation 
process  associated  with  the  award  milestones  is  very 
beneficial. 


17.  Software  Development  Cost  Control? 

a.  yes 

b.  Favorable  -  Use  cost  variance  on  end  item  as 
criteria 

c.  No  effect  -  did  not  distinguish  between  engine  and 
control  software. 

d.  -  See  16  - 


18.  Software  Delivery  Timeliness? 

a.  yes 

b.  May  be  unfavorable  -  Focused  schedule  attainment 

usually  degrades  software  quality 

c.  Yes  -  award  fee  effect  on  software  same  as  engine 
hardware . 

d.  -  See  16  - 
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19.  Software  Maintenance  Costs? 

a .  unknown 

b .  unknown 

c.  N/C 

d.  -  See  16  - 


20.  Software  Reliability? 

a.  yes 

b.  Award  fee  should  reward  robust  software  that  handles 

all  input  conditions  without  degrading  operating 
capability. 

c.  N/C 

d.  -  See  16  - 


21.  Software  Documentation? 

a.  yes 

b.  Put  CORL's  in  criteria 

c.  N/C 

d.  -  See  16  - 


22.  Software  Testability? 

a.  yes 

b.  To  much  emphasis  on  test  results  can  hurt  the 

overall  program  -*  l.e.  no  body  tests  till  they 
absolutely  have  to.  User  testing  of  delivered 
software  is  a  good  Indicator  of  job  done. 

c.  N/C 

d.  -  See  16  - 


23.  Software  Quality? 

a.  yes 

b.  Use  final  product  user  evaluation  for  criteria 

c.  N/C 

d.  -  See  16  - 


24.  In  addition  to  award  fee,  what  other  tool  does  a 
program  manager  have  to  control  the  software  development 
process? 

a.  SDCCR's 

b.  Personnel;  proper  dedicated  equipment;  detailed 
status  of  machine  code,  and  test;  tailored  MIL-STD- 
2167A,  get  Involved  positively  -  be  a  part  of  the 
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solution/can't  add  to  the  problem. 

c.  1)  Integrated  master  plan/schedule  on  monthly 
bases . 

2)  Costs  monthly. 

3)  Performance  measures  on  monthly  bases. 

4 )  Reviews . 

d.  Regularly  scheduled  technical  reviews  similar  to  PDR 
and  CDR.  This  Is  several  levels  above  a  TIM'S. 


25.  How  has  having  the  award  fee  on  your  program  benefltted 
the  overall  cost,  schedule,  and  performance? 

a.  Incentlvlzes  and  focused  the  contractor  and  the 
Government 

b.  Focused  attention  to  specific  areas  of  achievement. 

c.  1)  Good  motivator. 

2)  Not  the  thing  that  make  you  do  a  good  job. 

3)  Award  fee  Is  the  result  not  the  means. 

4)  If  the  contractor  does  what  he  plans  to  do  he 
receives  100%  fee  for  period.  My  success  as  a  good 
contractor  Is  to  do  what  you  said  you  are  going  to 
do.  CPAR  Is  another  motivator. 

d.  The  evaluation  process  has  been  beneficial  In 
managing,  cost,  schedule  and  performance  by  pointing 
out  strong  and  weak  points. 


26a.  Do  you  plan  to  have  an  award  fee  on  future  contracts? 

a .  yes 

b.  yes 

c.  yes 

d.  yes 


26b.  Why  or  why  not? 

a.  You  get  paid  for  good  performance 

b.  With  good  percentage  schedule  and  criteria  It  Is  a 
motivator . 

c.  Fair  contracting  approach 

d.  Because  I  believe  In  the  evaluation  process 


If  26a  Is  NO,  go  to  27. 

26c.  Will  the  award  fee  on  future  contracts  Include 

software  development? 

a .  yes 

b .  yes 

c.  yes 

d.  yes 
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26b.  Why  or  why  not? 

a.  Essential  element  of  program 

b.  Focus  attention  to  Important  areas 

c.  N/C 

d.  This  contractor  Is  directing  Its  goals  towards 

software  efforts  and  away  from  hardware  efforts 


27a.  Was  having  an  award  fee  on  your  progreun  a  hindrance  to 
the  overall  cost,  schedule,  and  performance? 

a.  no 

b.  no 

c.  no 

d.  no 


27b.  Why  or  why  not? 

a .  Management  tool 

b.  N/C 

c.  If  measure  was  milestone  the  milestone  maybe  dumb. 

This  was  not  held  against  them. 

d.  Because  all  our  efforts  were  not  based  on  $  amounts 
but  on  progreun  satisfaction  In  quality  of  the  end 
Item. 


27c.  If  the  award  fee  was  a  hindrance,  what  did  you  do  to 
overcome  It? 

a.  It  Is  beneficial  -  not  a  hlnderance 

b.  N/C 

c.  N/C 

d.  N/A 
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The  remaining  questions  are  optional  but  are  requested  to 
complete  the  research  effort. 

28.  Currently,  the  only  written  guidance  for  Award  Fee 
Plans  is  AFSC  "Award  Fee  in  Systems  Acquisition  -  A  Handbook 
for  Program  Directors  and  Contracting  Officers".  Please 
comment  on  attachments  2  and  3  and  answer  the  following 
questions . 


Please  comment  below  or  directly  on  the  attachments. 


28a. 

What  effect  on  the  process 

of  software  development 

would 

they  have? 

a. 

N/C 

b. 

N/C 

c. 

N/C 

d. 

The  award  fee  evaluation  criteria  establishes 
primary  ares  of  concern. 

28b. 

What  benefit  (or  hindrance) 

would  they  be  to  the 

program  office  and  the  contractor  if  made  part  of  the 
request  for  proposal  and  source  selection? 

a.  N/C 

b.  N/C 

c.  N/C 

d.  It  would  be  beneficial  because  it  would  give  the 
proposal  manager/program  manager  a  heads  up  to 
understand  what  customer's  major  concerns  are. 


28c.  Would  they  help  the  contractor  understand  what  must  be 
accomplished  to  receive  an  award  fee? 


a. 

N/C 

b. 

N/C 

c. 

N/C 

d. 

They  would  "summarize” 

the  major  points  only. 

Progreun  manager  should 
levels  of  concern. 

interpret  and  break  out  lower 
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Final  Comments  made  by  interviewees 

a.  As  examples,  the  attached  listing  is  O.K.;  however 
it  must  be  focused  on  the  total  progreun  not  just 
software  (unless  it  is  a  software  program)  or 
disciplines  (e.g.  systems  engineering  and  test). 

The  focus  must  be  on  products .  e.g.  Air  Vehicle, 
Training  System,  Total  Communications  System,  etc.. 
The  specific  criteria  in  the  attached  ares  are  very 
insightful  and  could  be  adapted  to  a  total  program 
focus  as  suggested  above. 

b.  1)  Too  many  award  fee  periods,  4  to  5  is  plenty 

2)  Criteria  should  be  very  program  specific  aimed 
at  attaining  desired  end  item  -  not  how  well  he 
checked  the  squares. 

3)  Contractor  receive  requirements  from  the  PCO  not 
Government  program  office. 

4)  The  award  fee  percentage  versus  ratings  is 
important  and  should  be  shown. 

5)  Tell  a  contractor  exactly  what  you  want  done  and 
grade  him  on  how  well  he  did  what  you  asked  for. 

6)  Research  and  Development  programs  must  have  more 
flexibility  to  change  and  adjust  to  unexpected 
program  demands  than  production  systems. 

c.  6  month  period  -  snapshot  of  next  2  or  more  events. 
Don't  over  monitor.  Need  to  focus  on  long  term 
goals  as  well  as  short  term  goals.  To  make  criteria 
better,  go  from  excellent  to  unacceptable  and  make 
good  spread  larger  40-79  points. 

d.  I  want  to  repeat  that  the  evaluation  process  is 
critical  at  the  progreun  level.  The  award  fee  based 
on  the  outcome  of  the  evaluation  process  will  be 
used  to  judge  the  program  manager  by  management. 
Praise  or  corrective  action  instituted  by  the  award 
percentage  than  determines  how  much  management 
"help"  is  supplied. 
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PERIOD  3  PERIOD  3  PERIOD  3  PERIOD  3  PERIOD  3  PERIOD  3 


F-22  WEAPON  SYSTEM  AND  F119  ENGINE 
AWARD  FEE  PLAN 


1  APRIL  1992 

(SopcEsedes  {dan  dated  24  Sqxanber  1991) 

EVALUATION  PERIOD  3 
1  APRIL  1992  -  30  SEPTEMBER  1992 


APPROVED: 


-r 

>jAMESA.FAlN,Jr. 

MbMor  General,  USAF 

Director  of  F>22  SPO 

Fo22  Award  Fee  Detenainiiig  Official 


PERIOD  3  PERIOD  3  PERIOD  3  PERIOD  3  PERIOD  3  PERIOD  3 
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F-22  WEAPON  SYSTEM  AND  F119  ENGINE 
AWARD  FEE  PLAN 


L  INTRODUCTION 


ThC^]CCificcnteEiftaild|BOOedllXCS  used  to  tnnnifnr  rjiinrarfnrpM-firtmigTwsff  and, 

tbexi^,  ncammeod  awaid  fise  p^ments  to  tfae  Fee  I>etennining  QfBctal  (FDO)  ate  desofi^ 
intfaisplan.  An  award  fee  is  med  to  motivate  ffltcBnmrperfhnnannft  by  the  conaactois  to 
execnte  the  P-22  and  F119Bagine<aing  and  MannfactnringDevdqanent  (HMD)  program. 
Uie  awarded  amotmt  is  detennined  by  the  Government’s  review  of  management  ^ 
performance  areas  ondatfae  control  of  the  contractors.  Tbe  F-22  and  F119  award  fee 
detenrunadonswffl  be  made  sqiaraiely  for  the  we^onqrstem  and  engine  contractors.  Award 
fix  detominations  made  by  die  Oovemment  are  not  snbjea  to  tbe  "Deanes,”  "Allowable  Cost 
andPtqrmeat,"  or  "Tenninadon"  danses  nfdie  conttacL 


n.  AWARD  FEE  INTEGRITY 


cfitWrartnrpiw^rirTnanr^  anrf  auwmt  fne  rfigihility  «  However.dtt 

process  is  qqilicitenoa^  to  allow  the  contractors  every  oppottuniy  to  understand  how  tto 
award  amount  is  based  on  perfenoiance.  Every  effort  be  made  the  Govemmem  to 

assmefemuasofevalnation,  as  wen  as  prompt  and  consistent  feedbadc.  Contractor 
perfonnanoe,  as  assessed  the  bnegrat^  Prodna  Teams  CIFTs)  and  fencdanals,  win  form 

the  basis  fer  award  fee  dUmrsements,  with  the  final  deterndnadan  made  by  die  FDO  for  both 
the  F-22  weqxm  qrstem  and  die  FI  19  engine  contractors. 

m.  DEHNinON  OF  TERMS  AND  RESPONSIBILmES 


A.  TbeFDO  for  die  P-22/F119  program  is  the  Program  Director,  ASD/YF,  whose 
teqwnsihilWes  are: 

1.  To  approve  the  award  fee  plan  and  anthorizeai^  changes  to  the  plan  throughout 
the  life  of  die  contract 


2.  To  rppro ve  die  members  of  the  Award  Fee  Review  Board  (ARB )  and  anioint  a 
Chairman. 


3.  To  Ajfwmirwt  dw  amount  of  award  fee  earned  andp^able  to  tbe  F-22  weqion 
system  and  FI  19  engine  connactors  for  each  evaluation  period. 

4.  To  notify  the  contractors  of  die  amount  of  fee  awarded  diem  at  die  final  evaluation 
eachperiod,  with  a  description  of  their  strengths  and  improvemem  items  which  impacted  die 
award  fee  dedskm  for  that  evaluation  pericid. 

B.  The  ARB  consists  ofa  chairman,  a  secretariat  a  recorder,  die  technical  ditecci>r» 
iqiresentatives  firom  the  IPTs  and  functional  staffs  in  die  System  Program  Office  (SPO), 
Director  of  Logistics  Support  at  the  Sacramemo  Air  Logistia  Center,  and  tbe  commanders  of  the 
Defense  Plant  Rqsesentarive  Offices  (DPROs)  at  die  prime  contractors.  TheARBandits 
merhers  ate  listed  in  Attachment  (Atdi)  1.  Tlie  responsibilities  of  die  ARB  are: 
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1.  To  review  the  monthly  tepons,  as  snbmiaed  by  the  IPTs^uncrionals. 

2.  Tn  fnnm/tnr^’  pmrfnrmanm  wcfng  the  TPT«yfrinrrinnal«' 

recommendations  and  any  other  peninent  infomiadon. 

3.  To  develop  the  ARB  recomnisndadon  on  the  zating  and  amonnt  of  die  award  £se. 

4.  To  recommend  diang^  to  die  Award  Fee  Flan. 


S.  To  independendy  provide  xegnlarpedbzmance&edbadc  to  their  individnal 
contractor  connteipaits,  qtedfically  hi^tli^hting  strengths  and  improvement  items,  per  die 

trritma  in  tfi^plan  anH  ««  tti  thm  tnnnthly  Igponjc,  At  ttw  miHtwm  nnri  final,  this 

firodback  sho^  occnr  prior  to  the  IFTs/fimcthinals  pneaenring  their  assessment  to  the  ARB. 


C  The  ARB  chairman,  the  Dqmty  Program  Director,  ASD/YF,  whose  reqionsibilines  ate: 


1.  To^ipointanARB  secretariaL 

2.  To  qiprove  die  selection  of  Petfonnance  Monitor  Focal  Points  (PMI?s). 

3.  To  provide  performance  feedbadc,  in  accordance  widi  the  areas  of  emphasis  and 
criteria  in  the  award  fix  plan,  at  a  top  level,  to  the  contractors  on  a  regular  basis. 

4.  To  condna  ARB  meetings  and  break  any  ries  in  ARB  voting. 

5.  To  review  recommended  dianges  to  the  Award  Fee  Plan  and  provide  the  FDO  with 
a  recommendation  on  wfaidi  dianges  shor^  be  made. 

6.  To  present  the  ARB's  recommended  rating  and  award  fee  range,  with  supporting 
data,  to  die  FDO. 


D.  The  ARRsec7-eianar,ASD/YFMPsAvmrd  Fee  Brandi,  whose  responsibilities  are: 

1.  To  consolidate  the  ARB's  assessment  and  recommendation  for  presentation  to  the 
FDO,  at  both  the  Tnniwriii  anrf  the  final  of  ^a<4i  period  of  petfotmance. 

2.  Tn  o»l«»rf  a  n«rnrTW'n/ho  win  tnafntawi  the  minntea  erf  the  ARB  mai^inga,  nnrily 

ARB  board  members  and  PMFPsvriien  the  monthly  tgions  and  the  midterm  and  final 
briefings  are  due,  distribate  forms,  receive  and  distrfoote  completed  monthly  rqiorts  to  aU 
directors,  and  perform  odier  dudes  as  assigned  by  the  ARB  chairmaiL 

3.  To  maintain  the  Award  Fee  Plan,  inclodiog  any  updates  as  qqaoved  by  the  ARB 
and  the  FDO. 


4.  To  maintain  die  award  fee  files,  including  current  copies  of  the  Award  Fee  Plan, 
any  internal  procedures,  the  PMFPs'  monthly  reports,  and  any  other  documentation  having  a 
bearing  on  ^  FDO's  dedsioa  * 

E.  The  PMFPs,  who  are  listed  in  Atch  2,  aa  as  fedlitattns  for  the  SPO  IPTs  and 
functionals  in  the  award  fee  area.  Their  responsibilities  are: 
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1.  To  condnnally  collect  award  fee  infonnadon  related  to  die  contractor's  petfoimance 
ftom  odier  members  of  the  SPO,  DPRO  members,  and  other  Govenimem  team  members  in 
a.«ignwi  ateas,  in  accordance  wiih  Atch  3. 

2.  Tn  pnwpam  mnnfhly  P<»rffWTnanf;e  Repnits  fnr  each  evalnatinn  period  in  die  format 

shown  in  Atdi  4  and  submit  dsese  reports  to  the  secretariat  not  later  than  8  waddng  afier 

tteendofeacbmoodL  These  moodily  reports  will  be  coordinated  dnon^  the  approisiaie 
IPT/M»fiinrtinnal  Airrtrurr^  anrf  grill  thft  fYiwrall  w^lnatifwi  pngtrinn  fifthat  OT 

fimcdonal 

3.  T<>  prffparw  midtBTTii  and  final  TPT/Rinrrinnal  hri^finga  fnrpBwmtarinn  tn  the  ARR 
Tln»^  firiafifigg  qim  lv«  twtnimatfA  thmngh  fhft  appropriate  TPT  or  ffancrional  diTTCrOT,  and  wffl 
reflect  die  overall  evaluation  position  of  that  IPT  <ff  fimmioDal  Staff 

4.  Tn  pminA*  aariatnnfle  tn  fh#>  ARB  SBeretariat  in  pqMiaiian  of  the  midierm  and 
final  APB  rrenmmnn^Minn  tn  the  FDO. 

3.  To  undetstaiid  the  contraaor  performance  requirements  necessary  to  achieve  eadh 
level  of  performance  as  defined  in  Atch  8. 

d.  To  recommend  dianges  to  the  platL 

F.  The  F-22IF119  DPRO  members  at  all  foor  primary  contractors,  whose  responsibiliries 
are: 


L.  To  particpate  on  die  IPTs  or  fancrionals,  observe  cpnaactorperfomiance,  and 
provide  cntrent,  accnrate  infotmation  and  assessments  on  comractor  performance  to  the  SPO 
and  contractor  on  a  regnlar  basis. 

2.  To  snbmit  moodily  award  fee  reports  in  the  format  shown  in  Atch  4,  not  later  than 
S  woddng  d^  afier  the  end  ^die  month,  to  the  secxetaiiat,  who  will  distribote  them 
tfatougbootthe  SPO  as  appropriate. 

3.  To  recommend  changes  to  the  plaiL 

G.  The  Performance  Monitors,  who  are  all  F-22  SPO  members  and  other  Government 
personnel  associated  widi  die  F<22  weqwn  system  or  F119  engine  program,  i^ose 
responsibilities  are: 

1.  To  pardcqnte  in  the  program,  and  prov^  current,  accnrare  information  and 
assessments  on  contractor  performance  on  a  regular  basis  to  the  qipropriate  PMFP. 

2.  To  recommend  dianges  to  the  plan. 


3.  To  maintain  open,  honest,  and  frequent  communications  with  the  contraaors. 


rv.  AWARD  FEE  EVALUA'nON  REQUIREMENTS 


A.  The  standard  award  fee  period  of  perfonnance  is  6  months  in  duratioo; 
however,  the  first  pedod  was  shortened,  running  from  contraa  award  to  the  end  of  die 
Govemmem’s  fis^  year.  Subsequem  periods  ate  aligned  with  the  stan  and  midpoints  of  the 
Gwenxmem’s  fiscal  year  and  align  the  award  fee  evaluations  with  the  contractors'  bonus 
plans.  The  schedule  of  evaluation  periods  is  in  Atch  6.  There  are  17  evaluation  periods  over 
the  life  of  the  program. 
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B.  Hie  awaid  fee  evdiudoa  win  cover  four  areas  of  enqibasiswiiidiieflecttfae  balanced 
qipioadi  desired  in  Older  to  ddiver  the  final  product  wMirngierificarion,  OP  cost  and  on 
schedule.  The  first  three  areas  are  im^callyrdated;  a  strength  or  inqvovement  item  in  oie  of 
those  three  areas  win  potend^yinqiact  die  odier  two.  The  fomdi  area  enqhasizes  the  odier 
items  of  coocem  to  the  Sro  fiir  a  qiecific  period. 

1.  Oi»iM«n  Ptngtxe  TnamA  Wi»«pfin  gytwn/Ptigin..  fiywm  TVwtripmi^ 

Z  OvendlSdiednlePetfiinnance 

3.  Ovexan  Cost  Control 

4.  Odier  Program  Cnraridciations 

C.  wlii«4i  rnrtt^  ^tartfiralty  fV>vWTWT>«it*a  anti  Ofhirh  am 

mhsets  nf  the  area*  ef  «nph««i<,  wfll  lv»  period  m  weegtaaiy-  These  COteiia 

win  fitnberespand  on  the  fbiir  areas  of  emphasis.  The  criteria  fior  die  cmxentperhxl  are  in 
Atdi7fi)rtheweq)onsysiemaiidAtA8fo^eogin&  Theoitetiafiirtbefinrthareaof 
enqihasis  are  listed  in  pdodty  ordet;  from  hi^KSt  to  lowest. 

O.  Each  IFT  and  functional  o£5ce  win  devdop  an  overaU  adjective  rating  based  on  their 

rthgftnmrifwM  at  hnriittiafniH«»»fHi  anrf«h«»final,  tw  a/wwrtimrw  mth  Tarings 

described  in  Atdi  9. 

E.  The  intfividoalcriieria  ratings  are  reviewed  and  imitated  fay  the  ARB,  which 
devdops  die  overan  indtvidnai  coDtractor  ratings  fiar  both  F‘22  weqson  sysrem  and  die  F119 
engine.  These  ratings  wffldien  be  used  as  recommenriarimis  to  the  PDO,  to  assist  his 
detennination  of  die  percentage  of  fee  eadi  contractor  may  earn  at  die  end  of  tte  period. 

F.  The  total  availaMe  award  fee  is  calculated  at  apprnaimatrfy  9  percent  of  die  total 
estimated  cost  at  oomract  award.  Ronover  (a  process  wfaerdqr  award  fe  not  earned  in  aprior 
period  my  be  added  to  a  later  award  fee  pod.)  win  gygany  not  be  practiced  unless  die 
Govenmentdiooses  to  emphasireapetfiitmance  period  by  roHipg  the  unearned  award  fee 
forward.  Award  fee  amounts  which  aw  not  roUedfiatward  are  untannaMe,  TheGovanment 
reserves  die  right  to  diange  availaUe  award  fee  prior  to  the  begmning  of  any  pacncolar  period. 

V.  AWARD  FEE  ADMINI5TRATTVE  PROCEDURES 

A.  Comractor  Procedures: 

1.  The  conttactots  have  die  option  of  submitting  a  self-assessment  pyer,  not  to 
exceed  10  pages  in  length,  to  the  ARB  secretariat,  not  la^  than  5  wod^  d^  ate  the 
midterm  or  &al  date.  Theoonttactotsalsohavetheoptitsiofpresentiiigas^'assessmeiit 
briefing  to  the  ARB  not  later  than  10  working  dys  ate  the  midterm  and  final  date. 

Z  The  self-assessmempyer  win  be  submitxed  to  the  FDO  in  unedited  form  iqr  the 
ARB,  along  with  daiARB*sGovertnnent4)ased  assessments  and  tecommeaidations. 

3.  The  conttacton  win  submit  pn^sed  criteria,  within  each  area  of  emphasis,  as  wdl 
as  other  plan  changes,  for  the  next  period  of  perfoxmance,  as  requesmd  by  the  secretariat  to 
snppoit  scheduled  ARB  meetings. 
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B.  Government  Pioceduzes: 


1.  Theiecoider  will  notify  board  member  and  PMFP  IS  wotidng  days  before 

the  end  oftfae  evaluation  period  and  suspense  the  IFTs/fimctioDals' final  bnefing.  (See 
sdiednle  of  activiiies  at  Atch  10.) 

2.  The  PMFFs,  in  concert  with  tfaeir  director,  will  consolidate  tfaeirIPTs*/fBnctionaLs' 

final  hrinfing^  USfOg  infai'iiiarinn  3$  dOCnmeOted  ffl  tbe  mODthly  rqWSB,  ffpr  pmannfarinn  tO  the 
ARB  not  later  than  10  working  (Leafier  the  end  of  die  penod.  An  electronic  copy  of  the 
briefing  will  be  provided  to  the  secretariat  by  die  dose  of  die  ARB  briefing. 

3.  Hie  ARB  win  evalnate  die  IFT/functional  briefings  and  determine  the 
recommended  amoant  of  fim. 

4.  The  secretariat  will  consolidate  the  ARB*s  assessment  and  rrmmmmdatinn  for 
jinwifarinn  to  the  FDO  not  moTB  than  15  walking  d^  afierdie  dose  of  the  evalnation 
period. 


5.  The  FDO  win  determine  die  amonnt  of  award  fise  to  be  paid  to  both  the  F-22 
weapon  system  and  the  FI  1 9  engine  contractors  not  more  than  2S  working  da^  afier  the 
dose  of  tte  evahiatian  period. 

6.  A  mrtrrarf  mn^ifinafion  will  he  to  implement  the  FDfl  HefwniinaBQn  for 

eadi  contractar  not  more  dian  5  working  days  after  die  FDO  decision. 

7.  The  Government  win  provide  the  contractors  a  ddsuf  in  a  timdy  manner  at 
die  end  of  eadi  award  fee  detenninatkm. 

VL  MIDTERM  EVALUATION 

A.  ThnTPTii/lWrinnalaagiTl  pi-rthrmmiAgnw  enntrarfnr  evalnarinna  fhr  nach  evalliarinn 
period.  Tte  purpose  of  the  midterm  evaluation  is  to  advise  the  conaacnas  of  strengths  w 
performaioe  widdi  sfaaald  be  continned,  as  wen  as  inqnovementitems  winch  codd  result  in  a 
lower  raring  at  the  end  ofthe  evalnation  period.  This  midterm  evahiarion  win  not  result  in  arqr 
fee  be^  awarded,  but  win  be  an  irgmt  into  the  final  determinarian  fiv  the  evaluation  period. 

1.  The  recorder  win  notify  eadi  PMFP  IS  working  days  before  the  mid^int  ofthe 
evaluation  period  and  suspense  die  IPT/fimctional  midtenn  briefings. 

2.  MidtermbtiefingswiUbepresentedtothe  ARB  not  more  than  10  working  days 
after  die  midterm  ofthe  evaluation  period. 

3.  The  ARB  win  review  die  IPT/functional  briefings  and  determine  their  overaU 
recommendation  for  presentation  to  the  FDO.  The  FDO  wiU  approve  a  midterm  evaluation 
debrief  to  the  contractors. 

» 


5 


171 


6 


172 


AWARD  FEE  REVIEW  BOARD  MEMBERS 


AhwTWtfiaMitinan 

Membes 


*SeaetiEiat 

^Recorder 


F-22  Dqmty  ftogram  Dnector 
F-22  Dktctar  of  Projects 
F-22  Tecfanical  Director 

F-22  System  Rogtam  OfBce  Fractional  Directors 
SPO  Imitated  Prodna  Team  Directors 
DPROCommander5at  I,oddieedaiidPtatt  A  Whitney 
Director  of  Logistics  SiqjpoR  at  Sacramento  Air 

ASD/YFMP  (Award  Fee  Branch) 

Assistant  Award  Fee  Program  Manager 


*  Noitvodng  members 
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PERFORMANCE  MONITOR  FOCAL  POINTS 


Air  Vehide  Tniegraied 
PxoduaTeamCQT) 

CqxJmCotbia 

EngtnelPT 

Saiwti  Stimnow 

TnniinglPT 

C^  Jesse  James 

SiqipoftlFT 

M5  fVninw  Pulrm 

Engineering 

Lt  Cd  Steven  Obsi 

Program  Coatrol 

Ms  Carol  Geisen  (Wei^xm  System) 
Ikb  Sandy  Oibiqr  (Bogw) 

Comacts 

Lt  Col  Gary  Boyian 

Program  Mnagement 

KfrRflnDetmer 

Teat 

Lt  Col  Mark  Smbben 
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INSTRUCTIONS  FOR  INTEGRATED  PRODUCT  TEAMS  (IPTs)  AND 
FUNCTIONAL  OFFICES 


A.  Monitoring  and  Assessing  Perfbnnance. 

1.  IPT^fimcdonals  win  maintain  a  candnnonswciiteaiecaRi  of  the  contzactois'pexfaniuBice, 
indwiing  a^ois  fioD  odier  Govenaneot  petsomiel.  in  tfaeir  evabutioa  aiBi(s)  of  lespmsibiliQr. 
The  comnctoa'petfiaiMnce  win  be  assessed  baaed  on  coigqpoodence,iepons,  data  items, 
meetings,  and  conversarionstriiididemonsaateda6cooBaaors*day-to-diypeifionnapee.  A 

qiTttTTHtry  r/mrarmrt'  pw^nimanffw  otTI  >v»  |.n»p!«nw<<  WHYnriily  anH  Avjmt-nted  in  the 

Perfamumce  Repast.  These  moodilb^iqponswin  be  coordinated  tfaiongb  the 

fimcrinnaltfiiectar  and  win  aeflea  die  oveEaUevahiation  position  of  diat  team  or  fnnffionaL 

Z  brihmaligfvifrig,  nnrm«itw«inadhyfhg!OiirTgtafTafinth«»)wwirifiM>filea,nsedin 
pnqMmgfh#i«wrin«rinnTiipr»ff  mm  KeTg«agwH>ythaPi»rfr»TnanceMftmtnr  Focal  Points 
(PMFPs)  for  6  momfas  after  die  period  is  comideted,  in  order  mstqipottaiyiiiqniries  made  by 
die  Fee  Detemaaiag  OfSdal  (FDO). 

3.  IFTs^bnctionals  win  condna  an  assessments  in  an  open,  objecdye,  and  cooperate 
qBijt,sodigaflriraDdaccotateevaliiarinnis<d>tained.  Tbe  IPTs/fimcdonaLs  win  continue  to 
provide  tbeirconnteqpans  feedback  on  die  program  issnes  as  a  pan  of  the  day-to-dy 
management  of  die  program,  as  wen  as  fbmalfeedhadc  on  a  xegnlar  basis.  Tbe 
IPTs/fimcdonals  win  make  eveQT  efibn  to  be  oonsisient  fiom  poiod  to  period  in  tbeir  q)pic^ 

m  raring^  Thtcgnlt  wihanr^  t-ntiinu’mi  infnnuarinn 

fiomwfaicfamplanmgsoveiiientsinpetfbzxnanoe.  Positive  perfbnnance  accomplirinnents 
afaonid  be  empfetsmed  just  as  leadfly  as  negative  ones. 

B.  PetfonnanoeRqion. 

1. 11ielPTil^bnaionaIswinpRpaiemond]fyPecfi»manceRqionsasriiowninAtch4. 
These  wponssfaotddindndeHyot  ten  ^^icablemembec  of  the  System  ftognmOfBce 
(SPO),  Defense  Plant  Rqsesemadve  OfBtt  (DFRO),  or  atqr  odwr  Govenanent  agency  wfaidi 
m^h^idewmntiniinnMtionondmcosigactop'peribnnance 

Z  Moabfy  iqxws  wfll  be  snbniinnd  tfaroogfa  the  IPT  or  fancrional  director,  to  the  Award 
Fee  Review  Boaid  (ARB)  sectemiiat  on  tbe  ei^th  woddng  dqr  of  tbe  montb,  a^  win  be 
conridered  as  inpat  to  die  miAenn  and  final  1^/fancrinnal  briefing  to  the  ARB.  Moodily 
iqiorts  dmald  oontain  the  ftdlowing  mfinmarion: 

a.  The  evahiatot's  name  and  diat  of  the  IPT/ftmcrional,  the  award  fee  period,  the  month 
coveted  by  the  wpoiit,  the  coBnacwr  being  evalnyed,  and  ay  ^ledal  conditions  which  n^ 
have  ioflneoced  Aar  month's  perfinmance.  Special  conditions  sboold  consider  the  technical, 
economic,  and  scbednle  environment  tinder  which  die  conaactois  were  leqirited  tu  pesftnm. 

What  effea  did  the  environment  have  on  the  conttacrois’petftinnance? 

b.  The  oomactois' major  strengths  and  improvement  items  relative  to'planned 
acrivines  dniiiv  dm  mondi,  lismd  in  priority  <»ler,  indnding  the  ioqiaa  of  that  soengAtar 
improvement  item  on  tbe  execnrioo  of  the  ovetallprogtam.  ffqplicable,iDdudeiniheiinpaa 
aty  recommended  or  ongouigconective  actions.  Giwexsmplesofcontticion'petfonnance 
far  each  snength  and  improvement  item  listed.  Ratings  sboold  generally  be  besni  on  more 
than  one  ooconenoe,  not  sm^  incidents.  However,  severs!  minor  examples  miy  be  used  to 
si^pott  a  fenetdized  strength  or  inqxovement  item.  Rqwning  items  re^iiringinqsovement 
is  eqpeciillyimpottant  when  igpeamdcomnnaricarion  with  the  concactorfittlsm  resolve  die 
issue. 
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c.  AiecomniCTvird  adjective  rating  for  each  critertt  for  that  nuaafa.  The  strengths  and 
mptovenent  items  ahooldpriw^  concrete  examples  of  the  conttacton’perfonnancewhidi 
sqjpoR  the  xeconunetided  ndng. 

<L  The  strengths  end  improvement  items,  amotated  by  snssterMc  or  specifically  listed 
in  a  cover  leoet;  wiiaA  ate  leoommended  fiar  (Ssctsskn  dating  the  feedback  sessiaos  between 
the  DepoQT  PiugcBn  Director  and  his  i'Kim*»p«it.t 

C  Briefitigs. 

.1.  The  midtetm  and  final  briefings  will  be  presented  to  the  ARB  not  mote  than  10  woridng 
di^  after  ihftmidinm  or  close  of  die  evaliiarion  period,  and  will  contain  the  sane  9pe  of 
infiwniarion  as  fijond  in  the  months  lepoit. 

a.  Th*.  AnnM  a««waa  nvuT^n  TPT/fiinrri«Ml  r»f  rty.  fnntrartnr’c 

progiess  toward  ddiraing  the  end  product  widiitt  the  spedficarion,  cost,  and  scfaednle 
leqniiemens,  as  tdated  to  the  areas  of  emidiasis  and  criteria. 

b.  The  recommended  at^ective  taring  sbmild  be  based  on  the  eadreevalnation  period. 

2.  IFTs/ftmcrionals  will  be  nqiated  to  present  briefiogs  to  the  ARB  and  roo  at  tbe 
•audtetmaadhnalasieqaired.  n>T^|AfmMrrinn«l«  mtt  at«n  pnmwifa  a—imni’^  in  pnipting  fK» 
ARB  tecomnMatiiiatinn&rpiesentation  to  the  FDO  far  bodtthrimidtnnn  and  fi^  reviews  as 
required  by  the  secretariat. 

D.  Feedback. 

L  OBamotabiybasis,diB  IPTr/fimcriotuiig  will  presetat  die  HPO  and  ARB  cfaainnan 

and  iimimwHm#**  iawtty  y  imjiiHir  In  fi|H»  rtiiWM  Basedoo 

lim  diTMinw^  lim  AB^  ^attl  i— ^  ajwwMiwjit  nn  tim  jmgCatBd  ftedback  tO  be 

given  to  dwcotmactOEL  The  ARB  chairman  and  lFTsfihtKtiotals(aDdasb-IFTsfiriheAir 
Vehide)wfllprovte  that  farmal  ftedback  to  their  corniMpanstelat^  to  obsewedpcrfbr- 
manoe  doting  that  inaemL  AfterdiefeedbadciacaBipieted,aiiiHnoBoidamfacrecordwillbe 
seat  to  the  secretariat  docnmetBing  ooomlerioD  of  the  feedback  mad  any  open  atrioos  as  a  result 
oftheassesaoum. 

2.  Thg  TPTf^mrfinnal"  <wivtn<«y  <l»«tdhaHrprinrtn  ami  atW  mifllMm  and 

final  hiiafin^a  tg  cnSOte  the  nnntrfftinfa  rfnariy  nndafatand  (he  aiieiHtietB. 

3.  Feedback  should  also  be  condncted  by  the  IPTs/ftmcrionals  after  the  FDO's  decision  to 
provide  conrimmgtfiriogtie  with  the  contractnn  on  die  resolis  of  the  assrssmenr  and  the  . 
SPCs  eapectatioos  for  tte  next  evaluation  period. 

E.  Qiai^todieFIan. 

1.  IPTi/fhncTtonals  shall  recommend  any  changes  to  the  Award  Fee  Plan  far  the  next 
period  ^articnl^evalnation  criteria),  as  so^ierMed  by  the  setawariai.  The  teams  wiD  also 
provide  atymiot^  fin  rBCommendedcbanya  to  the  secretariat  and  ARB. 

2.  Cootdinaiioa  and  approval  of  the  changes  in  the  SPOnd  with  the  DPROwiD  take 
place  after  the  takherm  and  before  the  final  evaburion.  Every  teasonriileatienqpt  will  be  made 
to  coottfiiaie  changes  widi  the  comiactao  prior  to  the  chan^  nddng  place. 
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AWARD  FEE  SENSITIVE  •  FOR  OFFICIAL  USE  ONLY 


PERFORMANCE  REPORT 
Name:  ‘Award  fee  period: 

Organization: 

Contracton  Report  Month: 

Special  conditions  which  influenced  this  month's  performance  (If  any): 

Criteria:  (hhist  be  listed  in  Atdi  7  or  8) 

Strengths  of  the  contractor's  performance  for  this  criteria  (with  specific  examples): 

Impact  of  this  strength  on  cxecntion  of  the  program: 

Improvement  items  in  the  contractor's  performance  for  this  criteria  (with  specific 
examples): 

Impact  of  this  improvement  item  on  cxecntmn  of  the  program: 
Recommended  ai^ective  grade  for  thb  criteria: 

(Page  Bleak  After  Each  Criteria 

Next  Criteria:  Q^hist  be  listed  in  Audi  7  or  8) 

Strengths  of  the  contractor's  performance  for  this  criteria  (with  specific  examples): 

Impart  of  thb  strength  on  cxecntiim  vt  the  program: 

Improvement  items  in  the  contractor's  performance  for  thb  criteria  (with  specific 
examples): 

Impact  of  thb  improvement  item  on  execution  of  the  program: 

Recommended  adjective  grade  for  thb  criteria: 

(Continue  for  each  criterion.  Donociqjeattbeinfoimaxiooattfaetopoftfaefiisipage 
(coonactor,  name,  period,  etc.)  on  oKfa  page.)  * 
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SAMPLE  ^KIEFING  FORMAT 


AMMO  f«tSMMM-MliamOttLUH  ONLY 
WrKHIlUAMOUnnMflM 


ItASC  I 

COmilACTOfl 


STRENGTHS: 


CRiTERIA 


•  Th»Cenir$etordUABCwhlehbnpaet»dth»pngnmty^ 


lUPROVByiENT  ITEMS: 

•  Vi»CantneterdldnotdoABCtrtUehlnipaei»dai»pmgnmty^ 


NOTES:  litfonnation  in  Kanes  to  be  pravfdad  fay  iPTt/hmctioiiai  I 
offioas  I 

CiHaria  as  Incfleatsd  In  Plan;  anserffaKta  par  chart  I 

Uaaadlaetivsrannss  (EX,  VG,G;  SAT,  UNSAT)  ^ 

Aanuio  «  awimva  -  KR  omeiAL  UMi  ONLY 


Pot  asnsance  Bang  this  fannat,  see  die  Award  Pee  SeoeiariaL 


Atch5 


178 


AWARD  fEE  EVALUATION  PERIODS 


TIk  awani  fee  period  of  pexfonnanoe  is  ^icaUy  6  momhs  in  dimticii;  however,  the  fixst 
period  >wn  be  Aoneoed  in  OBder  to  align  eviiln»ifin«Twih^yropiiaie  times  in  the 
Oafvenmeot's  fiscal  year  and  to  align  award  fee  evdnarions  vnih  the  aamactaa' end  of  year 
activiries  such  as  boim  plans. 


RIQD 

START 

MIDrrERM 

FINAL 

1 

Cootiact  Award 

30  Sep  91 

2 

1  Oct  91 

31  Dec  91 

31  Mar  92 

3 

l.^ir92 

31  Inn  92 

30  Sep  92 

4 

10ct92 

31  Dec  92 

31  Mar  93 

5 

l.^ir93 

31  Inn  93 

30  Sep  93 

6 

10a93 

31  Dec  93 

31  Mar  94 

7 

lApr94 

31100  94 

30  Sep  94 

8 

10ct94 

31Dec94 

31  Mar  95 

9 

l.^ir9S 

30Jim9S 

30  Sep  95 

10 

10a95 

31  Dec  95 

31  Mar  96 

11 

lA]v96 

30  Jon  96 

30  Sep  96 

12 

10ct96 

31  Dec  96 

31  Mar  97 

13 

lApr97 

30  Jon  97 

30  Sep  97 

14 

10ct97 

31  Dec  97 

31  Mar  98 

15 

1  Apr  98 

30  Jim  98 

30  Sep  98 

16 

10a  98 

31  Dee  98 

31  Mar  99 

17 

1  Apr  99 

15  Aug  99 

31  Dec  99 

Tbia  scfaedole  of  cvihimorB  allows  for  a  sfaongned  first  period,  wfaicfa  win  give  the 
connautoaanawaidfeeeyalMtionpriortDdecidhigonbonnsesforpetsomelandpriorto 
docng  out  financial  accounts  in  Deognber.  Afisr  the  dust  fiat  period,  die  year  will  be 
divided  into  two  periods,  eadi  year  baving  the  same  ^de:  October  tfatougfa  March  and  .^nil 
tfarough  SeptembCT. 
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WEAPON  SYSTEM  AWARD  FEE  AREAS  OF  EMPHASIS  AND  CRITERIA  FOR 

PERIODS 

(To  be  v^xiated  each  period) 


UtEA  OF  EMPHASIS: 
Development 


Overan  Progrem  toward  Integrated  Weapon  System 


•  CRl'lEKIONDl:  Devdopment  of  Weapon  System.  This  critexian  assesses  die  qoa^ 

ofdie  eflbroassociaSBdyriihdeyeioFBng  a  q)ecTfi<^rincinni|diant  weapon  system,  indnding 
thm  mnA  oif  fiHtpTnwnwwa  MgMM  Hie  pmgteSS  tOWard 

scfafevgiientofdieDesigp-to-Cost(DTQ  goals  and  the  ooniiaaut^scfBmseo  achieve  those 
gn«l«  thmngh  rim  «wpl<wn<^«ri/»»  /if  engt  twrfitrrinn  aerivitiw  anrf  ■anil  also 

becopsideied.  AiMinnnany.anevalnarionyrillbemadetoagessifaereqwpavenessofdie 

rrmtTwfnr fri  trwyirpr^w.  rliny  _  mntHtrwww  thfww  rhiingM^  tf  miih  rtw  wigine 

enniriirinr,  jwd  perform  tha  aMhirmai  wwfe  geaerMBd  ftem  tfae^^inaejmg  f!hmge  Pmposal 
(ECP)prace3S.  Innovative  managemeot  and  technical pnoesses  and  sofadiaos  to  adiieve 
vraqin  system  pesfismanoe  are  also  evalnated  in  this  criterion. 

•  CRITERIOND2:  System  Interfaces.  This  critaioa  assesses  die  identification  of  aU 

<yBliwii  im^rf«rw«_  anH  tuih^rmnt^  fn  nTl  Ttniwrfiiffft  finntml  TVOTim^rnr  anrf 

identification  and  hui^iatian  of  any  Govemment  Hnniriied  E(|n^pnient  necessaiy  to  ariueve 
the  mtepaedreeqiop  system  requirements. 

•  CRITERION  D3:  Management  of  SnbcontmctorEfiiinls.  Hus  criteria!  assesses  the 
integtadon  of  an  snbeaottaetoreffiins  to  soppot  die  ovezdl  impaled  vreapon  system 
devdapment,  indodiag  efEocts  to  d^ie,  comol  and  flow  down  the  pexfonnance  requirements 
neoessaty  for  adnevement  of  oontcact  spedficarion  requirements. 

iREA  OF  EMPHASIS:  Overafl  Scfaednie  Peiformancei 


•  GRIIERIONSl:  Schednie  Management  This  criterion  evahiaBa  the  camactor's 
prrihnnance  against  platmed  schedules.  Hie  assessment  wffl  encompass  the  aaegrarion  of  the 
IncegtaredMawerSAdnle  with  die  Cos^ScfaednleContml  System  (C/SCS),  inchiding  an 
assessment  of  the  validity  of  die  causes  fig  schednie  adjisonemi  necessary  to  meet  InregiaiBd 
Master  Flan  criteria  and  the  efibctxveness  of  sdteduls  recovery  plans.  Hie  evainadon  will  also 
messnre  the  ooattactot's  ability  to  identify  potendd  sdiednle  psoUems  eady  and  project  the 
impaa  of  nearrerm  schedule  dianges  on  long-tetm  events. 

4REA  OF  EMPHASIS:  Overall  Cost  Control; 

■  CRITERION  Cl:  Cost  Management  Hiis  criterion  evalnrees  the  camactor's  actual  cost 
petfimuanceconqiaiedtodieestahUshrdPetfonnanceMeasotememBasdineaseapressedin 
the  C/SCS  and  die  efEscrive  use  of  die  cost  control  system  in  die  dqr-to-di^  management  of 
the  program,  indndingevalnaring  the  impaa  of  variances  and  inqilementiiig  corrective  action 
plaimg  and  an  evainanonoftfae  contractor's  progress  toward  C^CS  reviews.  Thiscriterion 
also  evaluates  the  cost  management  of  sabcootxaaor  efGotts  and  dmdy  deydopmtet  of  revised 
estimates  of  program  costs.  Hie  assessment  also  mdades  the  quality  atxltimdhie&  of 
perfinmance  measurement  reports.  This  criterion  wiU  also  evaluate  the  counactot'seffims  to 
moniior,  track  and  pursue  the  active  management  of  ovethead  costs. 


•  CRITERION  C2:  Affordability  Analysis  Process.  This  ctiteiioa  assesses  the  proc^  of 
aQocating  DTC  goals  and  the  fisedbackto  the  Integrated  Product  Teams  (flPTs)  to  allow  timely 
cost  redtution  eobns.  This  criterion  also  assesses  the  management  of  the  Affordability 
Analysis  process. 
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•  CRITERION  01:  IFT  Philosophy.  This  cxiteiianevaliiates  the  contractor's  effective  and 
ose  of  die  IPT  philospphy  to  manage  the  program  on  a  daily  This  management 

iippTna/4i  inriw/i^  the  ose  of  30  integrated  to  decisians  anri  tradeofEs,  and  die 

nm  nf  timriy^  wthinj  amnng,  and  at  all  levria  ahnwa  and  bdow  die  iPTs. 


•  CRITERION  02:  Programs.  This  cnteticnevalnates  the  effective  integration 

and  impltmiwitiitinn  nf  infwgriiy  priigrajiia. 


•  CRIIERI0N03:  Software  Programs.  This  cntedonevalnates  the  piogiess  toward 

_  .a  _ * _ _ fj-  « _ _  _ - _  ^  *  - ? - 


Enviroument,  as  wdl  as  progress  on  the  Softwiae  Development  Plans,  The  evrinatinn  will 

innlndu  an  nf  t'lUitTxetnt*^  ^ffrirrg  m  dwtmlnp  an  jntwgrniM  nf  tnnla 


evahiate  the  deveiapment,  continnons  hoiaovemeot,  and  adboeoce  to  consistent,  team-wide 
software  devdopmemph^ 

•  CRITERION  04:  Contract  Change  Process.  This  cntenonevalnaies  the  contractor's 
comidianoe  with  the  reqoitemexits  of  die  contrBct  change  process,  incinding  responses  to 
Advanced  Change  Stnri^  Notices.  Contract  Qumgeft^sals,  and  Kngtneermg  Change 
Proposals.  The  evalnaoon  win  look  at  die  logical  orgaidzatim  of  die  proposals,  stqjportability 
of  cost  data,  condnct  of  "good  ftuifa"  negotiatiaos,  arid  coordination  of  issues  widi  the  System 
RogtamOfiSce,  Defense  Plant  Rqpresmtative  Offices,  and  Defense  Comtact  Audit  Agency. 

•  CRITERION  05:  Mauaganent/Technkal  Information  System  (MITIS).  This 

rritwrinn  fhw  pmgma^  tirmrA  rontimied  definaim  and  implemenlaiion  of  die  M/nS 


•  CRITERION  06:  Training  System  Approach.  This  osexirmevalnates  the  steps  being 
taken  to  ftmnalize  the  txainiiig  sj^stem  strategy,  especiany  plans  for  itfendfying  teaming  s3^stems 

subconnactors. 


•  CRITERION  07:  Small  Bnsiness/SmaD  Disadvantaged  Business  Programs.  This 
critenm  evalnates  the  contractor  on  his  management  of  and  innovative  qpioadies  toward 
meeting  janaii  5tnd  gmaTI  disadvantagcd  bosiness  goals  as  ondined  in  the  contract. 


ENGINE  AWARD  FEE  AREAS  OF  EMPHASIS  AND  CRITERIA  FOR 

PERIOD  3 

(To  be  updated  each  period) 

ABEA.QF  EMPHASISl  Qveratl  Progress  toward  Integrated  Engine  System 
Development 

•  CRITERION  Dl:  Development  of  the  Engine  System.  TOs  crittrion  assesses  the 
quality  of  the  effons  associated  with  develqring  a  spwificaiion  compliant  engiite  system, 
iiududing  the  identification  and  successful  icsohmonof  teqniicments  issres.  The  progress 
toward  ai^evement  of  the  Design-to^st  (DTC)  goals  and  the  contractoi  's  efforts  to  achieve 
those  goals  throng  the  implementation  of  reasonable  cost  reduction  activines  and  stndies  will 
also  be  considered.  Additionally,  an  evaluation  will  be  made  to  assess  the  responsiveness  ox 
the  contractor  to  incoipoiate  dianges,  coordinate  those  changes,  if  apptoptiate,  with  the  weapon 
system  contraaor,  and  petfoim  the  addMonal  work  generated  ffom  the  ^gineenng  Change 
Proposal  (ECP)  process.  This  criterion  will  also  evaluate  die  effective  integration  and 
impietnenfarirtn  of  frtfiigrTtypmgramg-  Innovative  management  and  technical  processes  and 
soludons  to  achieve  engine  system  perfonnance  are  also  evaluated  in  this  criterion. 

•  CRITERION  D2:  System  Interfaces.  This  criterion  assesses  the  identification  of  all 

system  interfaces,  use  of  the  Associate  Contractor  Wodring  Group,  development  and 
sd’ierence  to  all  Tnti^rfar^  Control  Doemnents,  anH  identificarion  integradon  of  any 

*^ent  Famished  Equ^ment  necessary  to  adiieve  the  integrated  weqxm  system 
requirements. 

•  CRITERION  D3:  Management  of  Subcontractor  Efforts.  This  criterion  assesses  the 
integnuion  of  all  subcontraaor  effotts  to  suppott  the  overall  imegrated  engine  development, 
including  effotts  to  define,  control  and  flow  down  the  perfonnance  requinemems  necessary  for 
achievement  of  contract  specification  requirements. 

AREA  OF  EMPHASIS;  Overall  Schedule  Performance; 

•  CRITERION  SI:  Schedule  Management  This  criterion  evaluates  the  conttactor’s 
perfonnance  against  planned  schedules.  The  assessment  will  encompass  the  integration  of  the 
Imegrated  Maner  Schedule  with  the  Cost/Schedule  Control  System  (C/SCS),  including  an 
assessment  of  the  validity  of  the  causes  for  schedule  adjustments  necessaiy  to  meet  Integrated 
Master  Plan  criteria  and  the  effectiveness  of  schedule  recovery  plans.  The  evaluation  will  also 
measure  the  contractor's  ability  to  identify  porenrial  schedule  problems  early  and  projea  the 
impaa  of  near-term  schedule  changes  on  long-tenn  events. 

AREA  OF  EMPHASIS:  Overall  Cost  Control; 

•  CRITERION  Cl:  Cost  Management  This  criterion  evaluates  the  contraaor's  actual  cost 
perfonnance  compared  to  the  es^lishcd  Pcifonnance  Measurement  Baseline  as  e:qpriessed  in 
the  C/SCS  and  the  effective  use  of  the  cost  control  system  in  the  day-to-day  management  of 
the  piogiam,  including  evaluating  the  impaa  of  variances  and  implementing  coirecnve  acdon 
planning.  The  contraaor's  progress  towmd  QSCS  validation  will  also  be  evaluated.  'Aus 
criterion  also  evaluates  the  cost  managemem  of  subcontraaor  efforts  and  timely  development 
of  revised  estimates  of  program  costs.  The  assessment  also  includes  the  quality  and  timeliness 
of  perfonnance  measuiemeiu  reports.  This  criterion  will  also  evaluate  the  contraaor's  effoits 
to  monitor,  track  and  puisne  the  active  managemem  of  oveihead  costs. 
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•  CRITERION  C2;  AfTordability  Analysis  Process.  This  criterion  assesses  the  process  of 
allocating  DTC  goals  and  feedback  to  the  Integrated  Product  Teams  (IPTs)  to  allow  tmely 
cost  reduction  efforts.  This  criterion  also  assesses  the  management  of  the  Affordability 
Analysis  process. 

AREAQF  EMPHASIS;.  Other  ProgramConsideradonsi 

•  CRITERION  01:  IPT  IMuIasopfay.  This  criterion  evaluates  the  contractor’s  effeedve  and 
fffRciiTTr  use  of  the  IPTphilosopfay  to  manage  the  program  on  a  daily  basis.  This  management 
approach  inrlndiig  the  use  of  an  integrated  approat^  to  malre  decisions  and  tradeoffs,  and  the 

rrf  timely,  rfirnmiTnirafinn  wirhin,  among,  and  at  all  levds  above  and  below  the  IPTs. 

•  CRITERION  02:  Contract  Change  Process.  This  criterion  evalnates  the  connactoi's 
compliauoe  with  the  requirements  of  the  contract  change  process,  including  responses  to 
Advanced  Change  Study  Notices,  Contract  Change  Proposals,  and  Rngineermg  Change 
Ptoposals.  The  evalnatitm  win  look  at  the  logical  organization  of  the  proposals,  snppoitability 
of  cost  'iiua,  condnet  of  "good  faith"  negotiations,  and  coordination  of  issues  wuh  the  S3rstem 
Program  Office,  Defense  Plam  Representative  Offices,  and  Defense  Contract  Audit  Agency. 

•  CRITERION  03:  ManagementfTeclmical  Information  S:ptem  (M/TB).  This 
criterion  evaluates  the  progress  towaid  continued  definition  and  implementation  of  die  IvfATS 
requirements,  including  the  Video  Teleconferencing  Center  c^abilxiy. 

•  CRITERION  04:  Small  Business/Small  Disadvantaged  Buriness  Programs.  This 
criierion  evaluates  the  contractor  on  his  management  of  and  innovarive  spproachc!?  toward 
meeting  small  and  small  disadvantaged  business  goals  as  outlined  in  the  contract. 
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RATING  DEFINTnONS 


The  Fee  Detenniaing  OfSckl  (FDO)  will  detennine  an  overall  rating  of  each  conaaaors' 
performance,  which  will  be  related  to  the  peicem  of  award  fee  paid.  Tlte  relarionshq?  between 
raring  and  award  payment  is  r  below. 


Bating 
Excellent 
Very  Good 
Good 

SatisfactQiy 

Unsadsfactoty 


Percent  of  Award  Fee 
91-100% 
71-90% 
51-70% 

1-50% 

0% 


The  following  definitions  des^ aibe,  in  general,  the  types  of  performance  which  will  lead  to  th. . 
vaiiocs  grades  assessed  by  the  Iritegrt^Prodna  Teams/fhncrionals  and  recommended  by  the 
Award  ^e  Review  Board  to  the  FDO.  These  standard  rating  definitions  will  be  ^^lied 
against  the  areas  of  evaluation  and  edtenain  order  to  provide  an  assessment  of  ea^  prime 
contractor’s  performance  and  resultant  award  fee  rating.  The  abiliry  to  deliver  a  quality  product 
is  defined  £c^  a  piodua  peispecswe,  while  the  management  and  communications  aicas  are 
defined  from  the  process  perspective.  The  intent  oftteF-22/F119  program  is  to  use  the  award 
fee  to  manage  the  overall  program  and  to  motivate  the  comractots  to  execute  the  program  as 
planned  Snccessful  execution  ofthe  program  will  maxnnize  the  award  fee  amoum  given  to 
the  contractois. 


Excellent: 


•  A  high  probability  exists  drat  a  quality  pnxiua  win  be  deUvered  on  rime  and 
(mb^get 

*•  Sdiolule  is  met  as  planned  (deviations  are  miixrr  and  have  no  impaa  on  overall 
program) 

••  Potential  cost  problems  ate  aggiessivdy  identified  and  resolved  eariy 

•  Managemoit  initiatives  are  extremely  cfGsctrve 

•  Coirununicarions  are  excquionally  open,  timely,  and  meaningful 

Very  Good: 

•  A  moderate  to  high  probability  exists  that  a  quality  produa  will  be  delivered  on  rime  and 
on  budget 

«•  Sdioiule  is  met  as  planiud,widi  minor  rescheduling  required  (deviations  are 
minor  and  have  Hide  impact  on  overall  program) 

•*  Potential  cost  problems  are  proactively  identified  and  resolved 

•  Management  initiatrves  are  highly  efiecrive 

•  Commutucations  are  consistently  open,  timely,  and  meaningfiil 

Good: 


•  A  moderate  probability  exists  that  a  quality  product  will  be  delivered  on  time 
and  on  budget 

••  Schedule  is  usuaUy  met  as  planned,  with  some  rescheduling  required 
••  Coniraaor  demonstrates  ability  to  identify  and  resolve  cost  problems 

•  Managemeiu  initiatives  are  usually  efiecrive 

•  Communications  are  usually  open,  timely,  and  meaningful 
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Satisfactory: 

■  Alowtomodenteprobabili^existstiiataqDali^piDdiictwiDbeddiveredantinieaiid 

onbudget 

*■  Srfv<Hnl«»  lifmatims  T^l«t»ping  atvt  ptngrwn  impaew  am  inrtiwtng 

—  More  aggiBggivB  a/^nw«  hy  «*»•  r^<mrtnr  ar^  tn  anrf  regnlve  eng 

problems 

•  Commimiations  are  somerimes  not  open,  ti^y,  and  meaningful 

« 

Unsadsfactoiy: 

•  A  low  probability  esisB  that  a  qoali^prodnawiD  be  delrveied  on  time  and  on 
budget 

-  SdwdoleamnoIiaTionBacMtent 

M  ^whilfty  «w  wWirtty  ti/t  ptnHwna  twqniww*  (imrtmmeatt  mtervanrinn 

•  iiiiiijrivM  aw»  meffwrt'iwBflriinnegwient 

•  fVwinniinirtifttM  >vft^«ti»w»ty  ff fH"g  ™  aivt  mftanmgfhlnagg 

No  award  fee  is  paid  for  onsarisfactoiypgfbmiance. 


» 


185 


AWARD  FEE  EVALUATION  SCHEDULE 


DAYS  SHOWN  ARE  WORKDAYS 


START 


(M) - 

M+5 


F-IS 

FINAL 

P-5 

<F) 

(6  months) 

Ff5 

PflO 

Ffl5 

Ff2S 

F*30 

»  3  MONTHLYREPOSTS  (DUE(»f  8THDAY) 

»  FEEDBACK  TO  OQNTSACTQKS 

»>  MIDTERM  BRIEFINGS  SUSFENSED 

■ 

>■  CON1RACTOR1S  MIDTERM  SELF-ASSESSMENT  PAPER  DUE 
■•REDBACK  TO  OONIRACTDRS 
>  MEXTERM  BRIEFING  DUE  TO  ARB 
»  ARB  RECOMMENDATION  TO  FDO 

»  FDO  NOTTFICATIQN  TO  CONTRACTOR  ON  MIDTERM 
PERFORMANCE 

•-FEEDBACK  TO  OCRITRACTORS 

-3  MONTHLY  REPORTS  (DUE  ON  8TH  DAY) 

-FEEDBACK  TO  CCRnRACTORS 

»  FINAL  BRILFINGS  SUSFENSED 

.  NOnnCATION  TO  CONTRACTOR  (»I  FLAN  CHANGES 
FOR  NEXT  PERIOD.  INCLUDING  CRHERU 

I 

-  CONTRACTOR'S  FINAL  SELF-ASSESSMENT  PAPER  DUE 
.  FEEDBACK  TO  CONTRACTORS 

-  FINAL  BRIEFINGS  DUE  TStARB 

-  ARB  RECOMMENDATION  TO  FDO 

-FDO  OETERMlNATK»I 
-FEEDBACK  TO  OONTRACTCEIS 
.CONTRACT  MCEXIFKATTON  ISSUED 
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